Linear_Regression/Multi-Variable Linear Regression.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Multi-Variable Linear Regession"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    "In this example, we will implement linear regression with multiple variables to predict the prices of houses.\n",
    "\n",
    "Suppose you are selling your house and you want to know what a good market price would be. One way to do this is to first collect information on recent houses sold and make a model of housing\n",
    "prices.\n",
    "\n",
    "The file ex1data2.txt contains a training set of housing prices in Portland, Oregon. The first column is the size of the house (in square feet), the second column is the number of bedrooms, and the third column is the price of the house.\n",
    "\n",
    "    NOTE: The example and sample data is being taken from the \"Machine Learning course by Andrew Ng\" in Coursera."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# initial imports\n",
    "import numpy as np\n",
    "from matplotlib import pyplot as plt\n",
    "%matplotlib notebook\n",
    "import seaborn as sns\n",
    "\n",
    "# setting graph properties\n",
    "plt.rcParams['figure.dpi'] = 300  # setting figure dpi for better quality graphs\n",
    "plt.rcParams['figure.figsize'] = [10,8]\n",
    "sns.set(context=\"notebook\", style=\"white\")  # graph styling using seaborn\n",
    "%config InlineBackend.figure_format = 'svg'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# imports from my models designed for these examples\n",
    "from models.linear_regression import compute_cost, gradient_descent, compute_gradient, normal_equation\n",
    "from models.data_preprocessing import add_bias_unit, feature_normalize"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading and Displaying Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading data ...\n",
      "['House size' 'No. of bedrooms']\n",
      "[[2104.00    3.00]\n",
      " [1600.00    3.00]\n",
      " [2400.00    3.00]\n",
      " [1416.00    2.00]\n",
      " [3000.00    4.00]]\n"
     ]
    }
   ],
   "source": [
    "print('Loading data ...')\n",
    "\n",
    "data = np.loadtxt('data/ex1data2.txt', delimiter=',')\n",
    "X = data[:, :-1]  # 47x2\n",
    "y = data[:, -1, None]  # 47x1\n",
    "m = y.size  # 47\n",
    "\n",
    "# printing first 5 elements\n",
    "with np.printoptions(precision=2, floatmode='fixed', suppress=True, linewidth=150):\n",
    "    print(np.array(['House size', 'No. of bedrooms']))\n",
    "    print(X[:5, :])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature Normalization\n",
    "\n",
    "By looking at the values, note that house sizes are about\n",
    "1000 times the number of bedrooms. When features differ by orders of magnitude, first performing feature scaling can make gradient descent converge much more quickly.\n",
    "\n",
    "The feature normalize function implements:\n",
    "- Subtraction of the mean value of each feature from the dataset.\n",
    "- After subtracting the mean, additionally scale (divide) the feature values by their respective “standard deviations.”\n",
    "\n",
    "The standard deviation is a way of measuring how much variation there is in the range of values of a particular feature (most data points will lie within ±2 standard deviations of the mean); this is an alternative to taking the range of values (max-min).\n",
    "\n",
    "For example, inside feature_normalize, the quantity X(:,1) contains all the values of $x_{1}$ (house sizes) in the training\n",
    "set, so np.std(X(:,1)) computes the standard deviation of the house sizes.\n",
    "\n",
    "At the time that feature_normalize function is called, the extra column of 1’s corresponding to $x_{0} = 1$ has not yet been added to X.\n",
    "\n",
    "**Implementation Note: When normalizing the features, it is important\n",
    "to store the values used for normalization - the mean value and the standard deviation used for the computations. After learning the parameters\n",
    "from the model, we often want to predict the prices of houses we have not\n",
    "seen before. Given a new x value (living room area and number of bedrooms), we must first normalize x using the mean and standard deviation\n",
    "that we had previously computed from the training set.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Normalizing the features\n",
      "['House size' 'No. of bedrooms']\n",
      "[[ 1.00  0.13 -0.23]\n",
      " [ 1.00 -0.51 -0.23]\n",
      " [ 1.00  0.51 -0.23]\n",
      " [ 1.00 -0.74 -1.55]\n",
      " [ 1.00  1.27  1.10]]\n"
     ]
    }
   ],
   "source": [
    "# ----------------Feature Normalization -----------------\n",
    "print(\"Normalizing the features\")\n",
    "\n",
    "X, mu, sigma = feature_normalize(X)\n",
    "# mu is the mean value\n",
    "# sigma is the standard deviation\n",
    "\n",
    "# adding intercept term to X\n",
    "X = add_bias_unit(X)\n",
    "\n",
    "# Printing first 5 normalized data points\n",
    "with np.printoptions(precision=2, floatmode='fixed', suppress=True, linewidth=150):\n",
    "    print(np.array(['House size', 'No. of bedrooms']))\n",
    "    print(X[:5, :])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note: the intercept term $x_{0} = 1$ is not normalized**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Gradient Descent\n",
    "\n",
    "Previously, we implemented gradient descent on a univariate regression problem. The only difference in a univariate and multi-variate linear regression gradient descent is that there is one or more than one feature in the multi-variate data matrix X. The hypothesis function and the batch gradient descent update\n",
    "rule remain unchanged.\n",
    "\n",
    "### Update equation\n",
    "\n",
    "The objective of linear regression is to minimize the cost function:\n",
    "\n",
    "$$ J(\\theta) = \\frac{1}{2m}\\sum_{i=1}^{m} \\left( h_{\\theta}\\left( x^{(i)} \\right) - y^{(i)} \\right) ^{2} $$\n",
    "\n",
    "where the hypothesis $h_{\\theta}(x)$ is given by the linear model :\n",
    "$$ \n",
    "h_{\\theta}(x) = \\theta^{T}x = \\theta_{0} + \\theta_{1}x_{1}\n",
    "$$\n",
    "\n",
    "the parameters of our model are the $\\theta_{j}$ values. These are the values we will adjust to minimize cost J(θ). One way to do this is to use the batch gradient descent algorithm. In batch gradient descent, each\n",
    "iteration performs the update \n",
    "\n",
    "$\n",
    "\\theta_{j} := \\theta_{j} - \\alpha \\frac{1}{m} \\sum_{i=1}^{m} \\left(h_{\\theta} \\left(x^{(i)}\\right) -y^{(i)}\\right)x^{(i)}_{j}$  (simultaneously update $\\theta_{j}$ for all j)  \n",
    "\n",
    "With each step of gradient descent, our parameters $\\theta_{j}$ come closer to the\n",
    "optimal values that will achieve the lowest cost J(θ)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Implementation Note\n",
    "\n",
    "In the multivariate case, the cost function can also be written in the folowing vectorized form:\n",
    "\n",
    "$$\n",
    "J(\\theta) = \\frac{1}{2m} \\left( X\\theta - \\vec{y} \\right)^{T} \\left( X\\theta - \\vec{y} \\right)\n",
    "$$\n",
    "\n",
    "where, \n",
    "$$\n",
    "X = \\begin{bmatrix} \\dots & \\left( x^{(1)} \\right)^{T}  & \\dots \\\\\n",
    "\\dots & \\left( x^{(2)} \\right)^{T} & \\dots \\\\\n",
    "& \\vdots & \\\\ \n",
    "\\dots & \\left( x^{(m)} \\right)^{T} & \\dots \\end{bmatrix} \\space\\space\\space\\space\\space \\vec{y} = \\begin{bmatrix} y^{1} \\\\ y^{2} \\\\ \\vdots \\\\ y^{m} \\end{bmatrix}\n",
    "$$\n",
    "\n",
    "The vectorized version is efficient when you’re working with numerical\n",
    "computing tools like Octave/MATLAB/NUMPY. If you are an expert with matrix\n",
    "operations, you can prove to yourself that the two forms are equivalent."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running gradient descent ...\n",
      "Obtained theta is: [340410.91897274 109162.68848142  -6293.24735132]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Text(0.5, 1.0, 'Covariance Graph with appropriate learning rate')"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Created with matplotlib (https://matplotlib.org/) -->\n",
       "<svg height=\"282.717813pt\" version=\"1.1\" viewBox=\"0 0 381.3125 282.717813\" width=\"381.3125pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       " <defs>\n",
       "  <style type=\"text/css\">\n",
       "*{stroke-linecap:butt;stroke-linejoin:round;}\n",
       "  </style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 282.717813 \n",
       "L 381.3125 282.717813 \n",
       "L 381.3125 0 \n",
       "L 0 0 \n",
       "z\n",
       "\" style=\"fill:none;\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 39.3125 239.758125 \n",
       "L 374.1125 239.758125 \n",
       "L 374.1125 22.318125 \n",
       "L 39.3125 22.318125 \n",
       "z\n",
       "\" style=\"fill:#ffffff;\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"text_1\">\n",
       "      <!-- 0 -->\n",
       "      <defs>\n",
       "       <path d=\"M 31.78125 66.40625 \n",
       "Q 24.171875 66.40625 20.328125 58.90625 \n",
       "Q 16.5 51.421875 16.5 36.375 \n",
       "Q 16.5 21.390625 20.328125 13.890625 \n",
       "Q 24.171875 6.390625 31.78125 6.390625 \n",
       "Q 39.453125 6.390625 43.28125 13.890625 \n",
       "Q 47.125 21.390625 47.125 36.375 \n",
       "Q 47.125 51.421875 43.28125 58.90625 \n",
       "Q 39.453125 66.40625 31.78125 66.40625 \n",
       "z\n",
       "M 31.78125 74.21875 \n",
       "Q 44.046875 74.21875 50.515625 64.515625 \n",
       "Q 56.984375 54.828125 56.984375 36.375 \n",
       "Q 56.984375 17.96875 50.515625 8.265625 \n",
       "Q 44.046875 -1.421875 31.78125 -1.421875 \n",
       "Q 19.53125 -1.421875 13.0625 8.265625 \n",
       "Q 6.59375 17.96875 6.59375 36.375 \n",
       "Q 6.59375 54.828125 13.0625 64.515625 \n",
       "Q 19.53125 74.21875 31.78125 74.21875 \n",
       "z\n",
       "\" id=\"DejaVuSans-48\"/>\n",
       "      </defs>\n",
       "      <g style=\"fill:#262626;\" transform=\"translate(51.031307 257.616406)scale(0.11 -0.11)\">\n",
       "       <use xlink:href=\"#DejaVuSans-48\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"text_2\">\n",
       "      <!-- 50 -->\n",
       "      <defs>\n",
       "       <path d=\"M 10.796875 72.90625 \n",
       "L 49.515625 72.90625 \n",
       "L 49.515625 64.59375 \n",
       "L 19.828125 64.59375 \n",
       "L 19.828125 46.734375 \n",
       "Q 21.96875 47.46875 24.109375 47.828125 \n",
       "Q 26.265625 48.1875 28.421875 48.1875 \n",
       "Q 40.625 48.1875 47.75 41.5 \n",
       "Q 54.890625 34.8125 54.890625 23.390625 \n",
       "Q 54.890625 11.625 47.5625 5.09375 \n",
       "Q 40.234375 -1.421875 26.90625 -1.421875 \n",
       "Q 22.3125 -1.421875 17.546875 -0.640625 \n",
       "Q 12.796875 0.140625 7.71875 1.703125 \n",
       "L 7.71875 11.625 \n",
       "Q 12.109375 9.234375 16.796875 8.0625 \n",
       "Q 21.484375 6.890625 26.703125 6.890625 \n",
       "Q 35.15625 6.890625 40.078125 11.328125 \n",
       "Q 45.015625 15.765625 45.015625 23.390625 \n",
       "Q 45.015625 31 40.078125 35.4375 \n",
       "Q 35.15625 39.890625 26.703125 39.890625 \n",
       "Q 22.75 39.890625 18.8125 39.015625 \n",
       "Q 14.890625 38.140625 10.796875 36.28125 \n",
       "z\n",
       "\" id=\"DejaVuSans-53\"/>\n",
       "      </defs>\n",
       "      <g style=\"fill:#262626;\" transform=\"translate(85.672738 257.616406)scale(0.11 -0.11)\">\n",
       "       <use xlink:href=\"#DejaVuSans-53\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 100 -->\n",
       "      <defs>\n",
       "       <path d=\"M 12.40625 8.296875 \n",
       "L 28.515625 8.296875 \n",
       "L 28.515625 63.921875 \n",
       "L 10.984375 60.40625 \n",
       "L 10.984375 69.390625 \n",
       "L 28.421875 72.90625 \n",
       "L 38.28125 72.90625 \n",
       "L 38.28125 8.296875 \n",
       "L 54.390625 8.296875 \n",
       "L 54.390625 0 \n",
       "L 12.40625 0 \n",
       "z\n",
       "\" id=\"DejaVuSans-49\"/>\n",
       "      </defs>\n",
       "      <g style=\"fill:#262626;\" transform=\"translate(120.31417 257.616406)scale(0.11 -0.11)\">\n",
       "       <use xlink:href=\"#DejaVuSans-49\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\n",
       "       <use x=\"127.246094\" xlink:href=\"#DejaVuSans-48\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"text_4\">\n",
       "      <!-- 150 -->\n",
       "      <g style=\"fill:#262626;\" transform=\"translate(158.454977 257.616406)scale(0.11 -0.11)\">\n",
       "       <use xlink:href=\"#DejaVuSans-49\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-53\"/>\n",
       "       <use x=\"127.246094\" xlink:href=\"#DejaVuSans-48\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"text_5\">\n",
       "      <!-- 200 -->\n",
       "      <defs>\n",
       "       <path d=\"M 19.1875 8.296875 \n",
       "L 53.609375 8.296875 \n",
       "L 53.609375 0 \n",
       "L 7.328125 0 \n",
       "L 7.328125 8.296875 \n",
       "Q 12.9375 14.109375 22.625 23.890625 \n",
       "Q 32.328125 33.6875 34.8125 36.53125 \n",
       "Q 39.546875 41.84375 41.421875 45.53125 \n",
       "Q 43.3125 49.21875 43.3125 52.78125 \n",
       "Q 43.3125 58.59375 39.234375 62.25 \n",
       "Q 35.15625 65.921875 28.609375 65.921875 \n",
       "Q 23.96875 65.921875 18.8125 64.3125 \n",
       "Q 13.671875 62.703125 7.8125 59.421875 \n",
       "L 7.8125 69.390625 \n",
       "Q 13.765625 71.78125 18.9375 73 \n",
       "Q 24.125 74.21875 28.421875 74.21875 \n",
       "Q 39.75 74.21875 46.484375 68.546875 \n",
       "Q 53.21875 62.890625 53.21875 53.421875 \n",
       "Q 53.21875 48.921875 51.53125 44.890625 \n",
       "Q 49.859375 40.875 45.40625 35.40625 \n",
       "Q 44.1875 33.984375 37.640625 27.21875 \n",
       "Q 31.109375 20.453125 19.1875 8.296875 \n",
       "z\n",
       "\" id=\"DejaVuSans-50\"/>\n",
       "      </defs>\n",
       "      <g style=\"fill:#262626;\" transform=\"translate(196.595783 257.616406)scale(0.11 -0.11)\">\n",
       "       <use xlink:href=\"#DejaVuSans-50\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\n",
       "       <use x=\"127.246094\" xlink:href=\"#DejaVuSans-48\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_6\">\n",
       "     <g id=\"text_6\">\n",
       "      <!-- 250 -->\n",
       "      <g style=\"fill:#262626;\" transform=\"translate(234.73659 257.616406)scale(0.11 -0.11)\">\n",
       "       <use xlink:href=\"#DejaVuSans-50\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-53\"/>\n",
       "       <use x=\"127.246094\" xlink:href=\"#DejaVuSans-48\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_7\">\n",
       "     <g id=\"text_7\">\n",
       "      <!-- 300 -->\n",
       "      <defs>\n",
       "       <path d=\"M 40.578125 39.3125 \n",
       "Q 47.65625 37.796875 51.625 33 \n",
       "Q 55.609375 28.21875 55.609375 21.1875 \n",
       "Q 55.609375 10.40625 48.1875 4.484375 \n",
       "Q 40.765625 -1.421875 27.09375 -1.421875 \n",
       "Q 22.515625 -1.421875 17.65625 -0.515625 \n",
       "Q 12.796875 0.390625 7.625 2.203125 \n",
       "L 7.625 11.71875 \n",
       "Q 11.71875 9.328125 16.59375 8.109375 \n",
       "Q 21.484375 6.890625 26.8125 6.890625 \n",
       "Q 36.078125 6.890625 40.9375 10.546875 \n",
       "Q 45.796875 14.203125 45.796875 21.1875 \n",
       "Q 45.796875 27.640625 41.28125 31.265625 \n",
       "Q 36.765625 34.90625 28.71875 34.90625 \n",
       "L 20.21875 34.90625 \n",
       "L 20.21875 43.015625 \n",
       "L 29.109375 43.015625 \n",
       "Q 36.375 43.015625 40.234375 45.921875 \n",
       "Q 44.09375 48.828125 44.09375 54.296875 \n",
       "Q 44.09375 59.90625 40.109375 62.90625 \n",
       "Q 36.140625 65.921875 28.71875 65.921875 \n",
       "Q 24.65625 65.921875 20.015625 65.03125 \n",
       "Q 15.375 64.15625 9.8125 62.3125 \n",
       "L 9.8125 71.09375 \n",
       "Q 15.4375 72.65625 20.34375 73.4375 \n",
       "Q 25.25 74.21875 29.59375 74.21875 \n",
       "Q 40.828125 74.21875 47.359375 69.109375 \n",
       "Q 53.90625 64.015625 53.90625 55.328125 \n",
       "Q 53.90625 49.265625 50.4375 45.09375 \n",
       "Q 46.96875 40.921875 40.578125 39.3125 \n",
       "z\n",
       "\" id=\"DejaVuSans-51\"/>\n",
       "      </defs>\n",
       "      <g style=\"fill:#262626;\" transform=\"translate(272.877396 257.616406)scale(0.11 -0.11)\">\n",
       "       <use xlink:href=\"#DejaVuSans-51\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\n",
       "       <use x=\"127.246094\" xlink:href=\"#DejaVuSans-48\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_8\">\n",
       "     <g id=\"text_8\">\n",
       "      <!-- 350 -->\n",
       "      <g style=\"fill:#262626;\" transform=\"translate(311.018203 257.616406)scale(0.11 -0.11)\">\n",
       "       <use xlink:href=\"#DejaVuSans-51\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-53\"/>\n",
       "       <use x=\"127.246094\" xlink:href=\"#DejaVuSans-48\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_9\">\n",
       "     <g id=\"text_9\">\n",
       "      <!-- 400 -->\n",
       "      <defs>\n",
       "       <path d=\"M 37.796875 64.3125 \n",
       "L 12.890625 25.390625 \n",
       "L 37.796875 25.390625 \n",
       "z\n",
       "M 35.203125 72.90625 \n",
       "L 47.609375 72.90625 \n",
       "L 47.609375 25.390625 \n",
       "L 58.015625 25.390625 \n",
       "L 58.015625 17.1875 \n",
       "L 47.609375 17.1875 \n",
       "L 47.609375 0 \n",
       "L 37.796875 0 \n",
       "L 37.796875 17.1875 \n",
       "L 4.890625 17.1875 \n",
       "L 4.890625 26.703125 \n",
       "z\n",
       "\" id=\"DejaVuSans-52\"/>\n",
       "      </defs>\n",
       "      <g style=\"fill:#262626;\" transform=\"translate(349.159009 257.616406)scale(0.11 -0.11)\">\n",
       "       <use xlink:href=\"#DejaVuSans-52\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\n",
       "       <use x=\"127.246094\" xlink:href=\"#DejaVuSans-48\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_10\">\n",
       "     <!-- Number of Iterations -->\n",
       "     <defs>\n",
       "      <path d=\"M 9.8125 72.90625 \n",
       "L 23.09375 72.90625 \n",
       "L 55.421875 11.921875 \n",
       "L 55.421875 72.90625 \n",
       "L 64.984375 72.90625 \n",
       "L 64.984375 0 \n",
       "L 51.703125 0 \n",
       "L 19.390625 60.984375 \n",
       "L 19.390625 0 \n",
       "L 9.8125 0 \n",
       "z\n",
       "\" id=\"DejaVuSans-78\"/>\n",
       "      <path d=\"M 8.5 21.578125 \n",
       "L 8.5 54.6875 \n",
       "L 17.484375 54.6875 \n",
       "L 17.484375 21.921875 \n",
       "Q 17.484375 14.15625 20.5 10.265625 \n",
       "Q 23.53125 6.390625 29.59375 6.390625 \n",
       "Q 36.859375 6.390625 41.078125 11.03125 \n",
       "Q 45.3125 15.671875 45.3125 23.6875 \n",
       "L 45.3125 54.6875 \n",
       "L 54.296875 54.6875 \n",
       "L 54.296875 0 \n",
       "L 45.3125 0 \n",
       "L 45.3125 8.40625 \n",
       "Q 42.046875 3.421875 37.71875 1 \n",
       "Q 33.40625 -1.421875 27.6875 -1.421875 \n",
       "Q 18.265625 -1.421875 13.375 4.4375 \n",
       "Q 8.5 10.296875 8.5 21.578125 \n",
       "z\n",
       "M 31.109375 56 \n",
       "z\n",
       "\" id=\"DejaVuSans-117\"/>\n",
       "      <path d=\"M 52 44.1875 \n",
       "Q 55.375 50.25 60.0625 53.125 \n",
       "Q 64.75 56 71.09375 56 \n",
       "Q 79.640625 56 84.28125 50.015625 \n",
       "Q 88.921875 44.046875 88.921875 33.015625 \n",
       "L 88.921875 0 \n",
       "L 79.890625 0 \n",
       "L 79.890625 32.71875 \n",
       "Q 79.890625 40.578125 77.09375 44.375 \n",
       "Q 74.3125 48.1875 68.609375 48.1875 \n",
       "Q 61.625 48.1875 57.5625 43.546875 \n",
       "Q 53.515625 38.921875 53.515625 30.90625 \n",
       "L 53.515625 0 \n",
       "L 44.484375 0 \n",
       "L 44.484375 32.71875 \n",
       "Q 44.484375 40.625 41.703125 44.40625 \n",
       "Q 38.921875 48.1875 33.109375 48.1875 \n",
       "Q 26.21875 48.1875 22.15625 43.53125 \n",
       "Q 18.109375 38.875 18.109375 30.90625 \n",
       "L 18.109375 0 \n",
       "L 9.078125 0 \n",
       "L 9.078125 54.6875 \n",
       "L 18.109375 54.6875 \n",
       "L 18.109375 46.1875 \n",
       "Q 21.1875 51.21875 25.484375 53.609375 \n",
       "Q 29.78125 56 35.6875 56 \n",
       "Q 41.65625 56 45.828125 52.96875 \n",
       "Q 50 49.953125 52 44.1875 \n",
       "z\n",
       "\" id=\"DejaVuSans-109\"/>\n",
       "      <path d=\"M 48.6875 27.296875 \n",
       "Q 48.6875 37.203125 44.609375 42.84375 \n",
       "Q 40.53125 48.484375 33.40625 48.484375 \n",
       "Q 26.265625 48.484375 22.1875 42.84375 \n",
       "Q 18.109375 37.203125 18.109375 27.296875 \n",
       "Q 18.109375 17.390625 22.1875 11.75 \n",
       "Q 26.265625 6.109375 33.40625 6.109375 \n",
       "Q 40.53125 6.109375 44.609375 11.75 \n",
       "Q 48.6875 17.390625 48.6875 27.296875 \n",
       "z\n",
       "M 18.109375 46.390625 \n",
       "Q 20.953125 51.265625 25.265625 53.625 \n",
       "Q 29.59375 56 35.59375 56 \n",
       "Q 45.5625 56 51.78125 48.09375 \n",
       "Q 58.015625 40.1875 58.015625 27.296875 \n",
       "Q 58.015625 14.40625 51.78125 6.484375 \n",
       "Q 45.5625 -1.421875 35.59375 -1.421875 \n",
       "Q 29.59375 -1.421875 25.265625 0.953125 \n",
       "Q 20.953125 3.328125 18.109375 8.203125 \n",
       "L 18.109375 0 \n",
       "L 9.078125 0 \n",
       "L 9.078125 75.984375 \n",
       "L 18.109375 75.984375 \n",
       "z\n",
       "\" id=\"DejaVuSans-98\"/>\n",
       "      <path d=\"M 56.203125 29.59375 \n",
       "L 56.203125 25.203125 \n",
       "L 14.890625 25.203125 \n",
       "Q 15.484375 15.921875 20.484375 11.0625 \n",
       "Q 25.484375 6.203125 34.421875 6.203125 \n",
       "Q 39.59375 6.203125 44.453125 7.46875 \n",
       "Q 49.3125 8.734375 54.109375 11.28125 \n",
       "L 54.109375 2.78125 \n",
       "Q 49.265625 0.734375 44.1875 -0.34375 \n",
       "Q 39.109375 -1.421875 33.890625 -1.421875 \n",
       "Q 20.796875 -1.421875 13.15625 6.1875 \n",
       "Q 5.515625 13.8125 5.515625 26.8125 \n",
       "Q 5.515625 40.234375 12.765625 48.109375 \n",
       "Q 20.015625 56 32.328125 56 \n",
       "Q 43.359375 56 49.78125 48.890625 \n",
       "Q 56.203125 41.796875 56.203125 29.59375 \n",
       "z\n",
       "M 47.21875 32.234375 \n",
       "Q 47.125 39.59375 43.09375 43.984375 \n",
       "Q 39.0625 48.390625 32.421875 48.390625 \n",
       "Q 24.90625 48.390625 20.390625 44.140625 \n",
       "Q 15.875 39.890625 15.1875 32.171875 \n",
       "z\n",
       "\" id=\"DejaVuSans-101\"/>\n",
       "      <path d=\"M 41.109375 46.296875 \n",
       "Q 39.59375 47.171875 37.8125 47.578125 \n",
       "Q 36.03125 48 33.890625 48 \n",
       "Q 26.265625 48 22.1875 43.046875 \n",
       "Q 18.109375 38.09375 18.109375 28.8125 \n",
       "L 18.109375 0 \n",
       "L 9.078125 0 \n",
       "L 9.078125 54.6875 \n",
       "L 18.109375 54.6875 \n",
       "L 18.109375 46.1875 \n",
       "Q 20.953125 51.171875 25.484375 53.578125 \n",
       "Q 30.03125 56 36.53125 56 \n",
       "Q 37.453125 56 38.578125 55.875 \n",
       "Q 39.703125 55.765625 41.0625 55.515625 \n",
       "z\n",
       "\" id=\"DejaVuSans-114\"/>\n",
       "      <path id=\"DejaVuSans-32\"/>\n",
       "      <path d=\"M 30.609375 48.390625 \n",
       "Q 23.390625 48.390625 19.1875 42.75 \n",
       "Q 14.984375 37.109375 14.984375 27.296875 \n",
       "Q 14.984375 17.484375 19.15625 11.84375 \n",
       "Q 23.34375 6.203125 30.609375 6.203125 \n",
       "Q 37.796875 6.203125 41.984375 11.859375 \n",
       "Q 46.1875 17.53125 46.1875 27.296875 \n",
       "Q 46.1875 37.015625 41.984375 42.703125 \n",
       "Q 37.796875 48.390625 30.609375 48.390625 \n",
       "z\n",
       "M 30.609375 56 \n",
       "Q 42.328125 56 49.015625 48.375 \n",
       "Q 55.71875 40.765625 55.71875 27.296875 \n",
       "Q 55.71875 13.875 49.015625 6.21875 \n",
       "Q 42.328125 -1.421875 30.609375 -1.421875 \n",
       "Q 18.84375 -1.421875 12.171875 6.21875 \n",
       "Q 5.515625 13.875 5.515625 27.296875 \n",
       "Q 5.515625 40.765625 12.171875 48.375 \n",
       "Q 18.84375 56 30.609375 56 \n",
       "z\n",
       "\" id=\"DejaVuSans-111\"/>\n",
       "      <path d=\"M 37.109375 75.984375 \n",
       "L 37.109375 68.5 \n",
       "L 28.515625 68.5 \n",
       "Q 23.6875 68.5 21.796875 66.546875 \n",
       "Q 19.921875 64.59375 19.921875 59.515625 \n",
       "L 19.921875 54.6875 \n",
       "L 34.71875 54.6875 \n",
       "L 34.71875 47.703125 \n",
       "L 19.921875 47.703125 \n",
       "L 19.921875 0 \n",
       "L 10.890625 0 \n",
       "L 10.890625 47.703125 \n",
       "L 2.296875 47.703125 \n",
       "L 2.296875 54.6875 \n",
       "L 10.890625 54.6875 \n",
       "L 10.890625 58.5 \n",
       "Q 10.890625 67.625 15.140625 71.796875 \n",
       "Q 19.390625 75.984375 28.609375 75.984375 \n",
       "z\n",
       "\" id=\"DejaVuSans-102\"/>\n",
       "      <path d=\"M 9.8125 72.90625 \n",
       "L 19.671875 72.90625 \n",
       "L 19.671875 0 \n",
       "L 9.8125 0 \n",
       "z\n",
       "\" id=\"DejaVuSans-73\"/>\n",
       "      <path d=\"M 18.3125 70.21875 \n",
       "L 18.3125 54.6875 \n",
       "L 36.8125 54.6875 \n",
       "L 36.8125 47.703125 \n",
       "L 18.3125 47.703125 \n",
       "L 18.3125 18.015625 \n",
       "Q 18.3125 11.328125 20.140625 9.421875 \n",
       "Q 21.96875 7.515625 27.59375 7.515625 \n",
       "L 36.8125 7.515625 \n",
       "L 36.8125 0 \n",
       "L 27.59375 0 \n",
       "Q 17.1875 0 13.234375 3.875 \n",
       "Q 9.28125 7.765625 9.28125 18.015625 \n",
       "L 9.28125 47.703125 \n",
       "L 2.6875 47.703125 \n",
       "L 2.6875 54.6875 \n",
       "L 9.28125 54.6875 \n",
       "L 9.28125 70.21875 \n",
       "z\n",
       "\" id=\"DejaVuSans-116\"/>\n",
       "      <path d=\"M 34.28125 27.484375 \n",
       "Q 23.390625 27.484375 19.1875 25 \n",
       "Q 14.984375 22.515625 14.984375 16.5 \n",
       "Q 14.984375 11.71875 18.140625 8.90625 \n",
       "Q 21.296875 6.109375 26.703125 6.109375 \n",
       "Q 34.1875 6.109375 38.703125 11.40625 \n",
       "Q 43.21875 16.703125 43.21875 25.484375 \n",
       "L 43.21875 27.484375 \n",
       "z\n",
       "M 52.203125 31.203125 \n",
       "L 52.203125 0 \n",
       "L 43.21875 0 \n",
       "L 43.21875 8.296875 \n",
       "Q 40.140625 3.328125 35.546875 0.953125 \n",
       "Q 30.953125 -1.421875 24.3125 -1.421875 \n",
       "Q 15.921875 -1.421875 10.953125 3.296875 \n",
       "Q 6 8.015625 6 15.921875 \n",
       "Q 6 25.140625 12.171875 29.828125 \n",
       "Q 18.359375 34.515625 30.609375 34.515625 \n",
       "L 43.21875 34.515625 \n",
       "L 43.21875 35.40625 \n",
       "Q 43.21875 41.609375 39.140625 45 \n",
       "Q 35.0625 48.390625 27.6875 48.390625 \n",
       "Q 23 48.390625 18.546875 47.265625 \n",
       "Q 14.109375 46.140625 10.015625 43.890625 \n",
       "L 10.015625 52.203125 \n",
       "Q 14.9375 54.109375 19.578125 55.046875 \n",
       "Q 24.21875 56 28.609375 56 \n",
       "Q 40.484375 56 46.34375 49.84375 \n",
       "Q 52.203125 43.703125 52.203125 31.203125 \n",
       "z\n",
       "\" id=\"DejaVuSans-97\"/>\n",
       "      <path d=\"M 9.421875 54.6875 \n",
       "L 18.40625 54.6875 \n",
       "L 18.40625 0 \n",
       "L 9.421875 0 \n",
       "z\n",
       "M 9.421875 75.984375 \n",
       "L 18.40625 75.984375 \n",
       "L 18.40625 64.59375 \n",
       "L 9.421875 64.59375 \n",
       "z\n",
       "\" id=\"DejaVuSans-105\"/>\n",
       "      <path d=\"M 54.890625 33.015625 \n",
       "L 54.890625 0 \n",
       "L 45.90625 0 \n",
       "L 45.90625 32.71875 \n",
       "Q 45.90625 40.484375 42.875 44.328125 \n",
       "Q 39.84375 48.1875 33.796875 48.1875 \n",
       "Q 26.515625 48.1875 22.3125 43.546875 \n",
       "Q 18.109375 38.921875 18.109375 30.90625 \n",
       "L 18.109375 0 \n",
       "L 9.078125 0 \n",
       "L 9.078125 54.6875 \n",
       "L 18.109375 54.6875 \n",
       "L 18.109375 46.1875 \n",
       "Q 21.34375 51.125 25.703125 53.5625 \n",
       "Q 30.078125 56 35.796875 56 \n",
       "Q 45.21875 56 50.046875 50.171875 \n",
       "Q 54.890625 44.34375 54.890625 33.015625 \n",
       "z\n",
       "\" id=\"DejaVuSans-110\"/>\n",
       "      <path d=\"M 44.28125 53.078125 \n",
       "L 44.28125 44.578125 \n",
       "Q 40.484375 46.53125 36.375 47.5 \n",
       "Q 32.28125 48.484375 27.875 48.484375 \n",
       "Q 21.1875 48.484375 17.84375 46.4375 \n",
       "Q 14.5 44.390625 14.5 40.28125 \n",
       "Q 14.5 37.15625 16.890625 35.375 \n",
       "Q 19.28125 33.59375 26.515625 31.984375 \n",
       "L 29.59375 31.296875 \n",
       "Q 39.15625 29.25 43.1875 25.515625 \n",
       "Q 47.21875 21.78125 47.21875 15.09375 \n",
       "Q 47.21875 7.46875 41.1875 3.015625 \n",
       "Q 35.15625 -1.421875 24.609375 -1.421875 \n",
       "Q 20.21875 -1.421875 15.453125 -0.5625 \n",
       "Q 10.6875 0.296875 5.421875 2 \n",
       "L 5.421875 11.28125 \n",
       "Q 10.40625 8.6875 15.234375 7.390625 \n",
       "Q 20.0625 6.109375 24.8125 6.109375 \n",
       "Q 31.15625 6.109375 34.5625 8.28125 \n",
       "Q 37.984375 10.453125 37.984375 14.40625 \n",
       "Q 37.984375 18.0625 35.515625 20.015625 \n",
       "Q 33.0625 21.96875 24.703125 23.78125 \n",
       "L 21.578125 24.515625 \n",
       "Q 13.234375 26.265625 9.515625 29.90625 \n",
       "Q 5.8125 33.546875 5.8125 39.890625 \n",
       "Q 5.8125 47.609375 11.28125 51.796875 \n",
       "Q 16.75 56 26.8125 56 \n",
       "Q 31.78125 56 36.171875 55.265625 \n",
       "Q 40.578125 54.546875 44.28125 53.078125 \n",
       "z\n",
       "\" id=\"DejaVuSans-115\"/>\n",
       "     </defs>\n",
       "     <g style=\"fill:#262626;\" transform=\"translate(144.43625 273.022188)scale(0.12 -0.12)\">\n",
       "      <use xlink:href=\"#DejaVuSans-78\"/>\n",
       "      <use x=\"74.804688\" xlink:href=\"#DejaVuSans-117\"/>\n",
       "      <use x=\"138.183594\" xlink:href=\"#DejaVuSans-109\"/>\n",
       "      <use x=\"235.595703\" xlink:href=\"#DejaVuSans-98\"/>\n",
       "      <use x=\"299.072266\" xlink:href=\"#DejaVuSans-101\"/>\n",
       "      <use x=\"360.595703\" xlink:href=\"#DejaVuSans-114\"/>\n",
       "      <use x=\"401.708984\" xlink:href=\"#DejaVuSans-32\"/>\n",
       "      <use x=\"433.496094\" xlink:href=\"#DejaVuSans-111\"/>\n",
       "      <use x=\"494.677734\" xlink:href=\"#DejaVuSans-102\"/>\n",
       "      <use x=\"529.882812\" xlink:href=\"#DejaVuSans-32\"/>\n",
       "      <use x=\"561.669922\" xlink:href=\"#DejaVuSans-73\"/>\n",
       "      <use x=\"591.162109\" xlink:href=\"#DejaVuSans-116\"/>\n",
       "      <use x=\"630.371094\" xlink:href=\"#DejaVuSans-101\"/>\n",
       "      <use x=\"691.894531\" xlink:href=\"#DejaVuSans-114\"/>\n",
       "      <use x=\"733.007812\" xlink:href=\"#DejaVuSans-97\"/>\n",
       "      <use x=\"794.287109\" xlink:href=\"#DejaVuSans-116\"/>\n",
       "      <use x=\"833.496094\" xlink:href=\"#DejaVuSans-105\"/>\n",
       "      <use x=\"861.279297\" xlink:href=\"#DejaVuSans-111\"/>\n",
       "      <use x=\"922.460938\" xlink:href=\"#DejaVuSans-110\"/>\n",
       "      <use x=\"985.839844\" xlink:href=\"#DejaVuSans-115\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"text_11\">\n",
       "      <!-- 0 -->\n",
       "      <g style=\"fill:#262626;\" transform=\"translate(22.81375 240.818361)scale(0.11 -0.11)\">\n",
       "       <use xlink:href=\"#DejaVuSans-48\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"text_12\">\n",
       "      <!-- 1 -->\n",
       "      <g style=\"fill:#262626;\" transform=\"translate(22.81375 207.711721)scale(0.11 -0.11)\">\n",
       "       <use xlink:href=\"#DejaVuSans-49\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"text_13\">\n",
       "      <!-- 2 -->\n",
       "      <g style=\"fill:#262626;\" transform=\"translate(22.81375 174.605081)scale(0.11 -0.11)\">\n",
       "       <use xlink:href=\"#DejaVuSans-50\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_4\">\n",
       "     <g id=\"text_14\">\n",
       "      <!-- 3 -->\n",
       "      <g style=\"fill:#262626;\" transform=\"translate(22.81375 141.49844)scale(0.11 -0.11)\">\n",
       "       <use xlink:href=\"#DejaVuSans-51\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_5\">\n",
       "     <g id=\"text_15\">\n",
       "      <!-- 4 -->\n",
       "      <g style=\"fill:#262626;\" transform=\"translate(22.81375 108.3918)scale(0.11 -0.11)\">\n",
       "       <use xlink:href=\"#DejaVuSans-52\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_6\">\n",
       "     <g id=\"text_16\">\n",
       "      <!-- 5 -->\n",
       "      <g style=\"fill:#262626;\" transform=\"translate(22.81375 75.285159)scale(0.11 -0.11)\">\n",
       "       <use xlink:href=\"#DejaVuSans-53\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_7\">\n",
       "     <g id=\"text_17\">\n",
       "      <!-- 6 -->\n",
       "      <defs>\n",
       "       <path d=\"M 33.015625 40.375 \n",
       "Q 26.375 40.375 22.484375 35.828125 \n",
       "Q 18.609375 31.296875 18.609375 23.390625 \n",
       "Q 18.609375 15.53125 22.484375 10.953125 \n",
       "Q 26.375 6.390625 33.015625 6.390625 \n",
       "Q 39.65625 6.390625 43.53125 10.953125 \n",
       "Q 47.40625 15.53125 47.40625 23.390625 \n",
       "Q 47.40625 31.296875 43.53125 35.828125 \n",
       "Q 39.65625 40.375 33.015625 40.375 \n",
       "z\n",
       "M 52.59375 71.296875 \n",
       "L 52.59375 62.3125 \n",
       "Q 48.875 64.0625 45.09375 64.984375 \n",
       "Q 41.3125 65.921875 37.59375 65.921875 \n",
       "Q 27.828125 65.921875 22.671875 59.328125 \n",
       "Q 17.53125 52.734375 16.796875 39.40625 \n",
       "Q 19.671875 43.65625 24.015625 45.921875 \n",
       "Q 28.375 48.1875 33.59375 48.1875 \n",
       "Q 44.578125 48.1875 50.953125 41.515625 \n",
       "Q 57.328125 34.859375 57.328125 23.390625 \n",
       "Q 57.328125 12.15625 50.6875 5.359375 \n",
       "Q 44.046875 -1.421875 33.015625 -1.421875 \n",
       "Q 20.359375 -1.421875 13.671875 8.265625 \n",
       "Q 6.984375 17.96875 6.984375 36.375 \n",
       "Q 6.984375 53.65625 15.1875 63.9375 \n",
       "Q 23.390625 74.21875 37.203125 74.21875 \n",
       "Q 40.921875 74.21875 44.703125 73.484375 \n",
       "Q 48.484375 72.75 52.59375 71.296875 \n",
       "z\n",
       "\" id=\"DejaVuSans-54\"/>\n",
       "      </defs>\n",
       "      <g style=\"fill:#262626;\" transform=\"translate(22.81375 42.178519)scale(0.11 -0.11)\">\n",
       "       <use xlink:href=\"#DejaVuSans-54\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_18\">\n",
       "     <!-- Cost J -->\n",
       "     <defs>\n",
       "      <path d=\"M 64.40625 67.28125 \n",
       "L 64.40625 56.890625 \n",
       "Q 59.421875 61.53125 53.78125 63.8125 \n",
       "Q 48.140625 66.109375 41.796875 66.109375 \n",
       "Q 29.296875 66.109375 22.65625 58.46875 \n",
       "Q 16.015625 50.828125 16.015625 36.375 \n",
       "Q 16.015625 21.96875 22.65625 14.328125 \n",
       "Q 29.296875 6.6875 41.796875 6.6875 \n",
       "Q 48.140625 6.6875 53.78125 8.984375 \n",
       "Q 59.421875 11.28125 64.40625 15.921875 \n",
       "L 64.40625 5.609375 \n",
       "Q 59.234375 2.09375 53.4375 0.328125 \n",
       "Q 47.65625 -1.421875 41.21875 -1.421875 \n",
       "Q 24.65625 -1.421875 15.125 8.703125 \n",
       "Q 5.609375 18.84375 5.609375 36.375 \n",
       "Q 5.609375 53.953125 15.125 64.078125 \n",
       "Q 24.65625 74.21875 41.21875 74.21875 \n",
       "Q 47.75 74.21875 53.53125 72.484375 \n",
       "Q 59.328125 70.75 64.40625 67.28125 \n",
       "z\n",
       "\" id=\"DejaVuSans-67\"/>\n",
       "      <path d=\"M 9.8125 72.90625 \n",
       "L 19.671875 72.90625 \n",
       "L 19.671875 5.078125 \n",
       "Q 19.671875 -8.109375 14.671875 -14.0625 \n",
       "Q 9.671875 -20.015625 -1.421875 -20.015625 \n",
       "L -5.171875 -20.015625 \n",
       "L -5.171875 -11.71875 \n",
       "L -2.09375 -11.71875 \n",
       "Q 4.4375 -11.71875 7.125 -8.046875 \n",
       "Q 9.8125 -4.390625 9.8125 5.078125 \n",
       "z\n",
       "\" id=\"DejaVuSans-74\"/>\n",
       "     </defs>\n",
       "     <g style=\"fill:#262626;\" transform=\"translate(16.318125 148.05375)rotate(-90)scale(0.12 -0.12)\">\n",
       "      <use xlink:href=\"#DejaVuSans-67\"/>\n",
       "      <use x=\"69.824219\" xlink:href=\"#DejaVuSans-111\"/>\n",
       "      <use x=\"131.005859\" xlink:href=\"#DejaVuSans-115\"/>\n",
       "      <use x=\"183.105469\" xlink:href=\"#DejaVuSans-116\"/>\n",
       "      <use x=\"222.314453\" xlink:href=\"#DejaVuSans-32\"/>\n",
       "      <use x=\"254.101562\" xlink:href=\"#DejaVuSans-74\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_19\">\n",
       "     <!-- 1e10 -->\n",
       "     <g style=\"fill:#262626;\" transform=\"translate(39.3125 19.318125)scale(0.11 -0.11)\">\n",
       "      <use xlink:href=\"#DejaVuSans-49\"/>\n",
       "      <use x=\"63.623047\" xlink:href=\"#DejaVuSans-101\"/>\n",
       "      <use x=\"125.146484\" xlink:href=\"#DejaVuSans-49\"/>\n",
       "      <use x=\"188.769531\" xlink:href=\"#DejaVuSans-48\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"line2d_1\">\n",
       "    <path clip-path=\"url(#p881de1886d)\" d=\"M 54.530682 32.201761 \n",
       "L 56.056314 55.317542 \n",
       "L 57.581946 75.667922 \n",
       "L 59.107579 93.591884 \n",
       "L 60.633211 109.38563 \n",
       "L 62.158843 123.308173 \n",
       "L 63.684475 135.586171 \n",
       "L 65.210108 146.418099 \n",
       "L 66.73574 155.977871 \n",
       "L 68.261372 164.417974 \n",
       "L 69.787004 171.872186 \n",
       "L 71.312637 178.457941 \n",
       "L 72.838269 184.278378 \n",
       "L 74.363901 189.424132 \n",
       "L 75.889533 193.974883 \n",
       "L 77.415166 198.000721 \n",
       "L 78.940798 201.563321 \n",
       "L 80.46643 204.716977 \n",
       "L 81.992063 207.50951 \n",
       "L 83.517695 209.983048 \n",
       "L 85.806143 213.175253 \n",
       "L 88.094592 215.839582 \n",
       "L 90.38304 218.064967 \n",
       "L 92.671488 219.925132 \n",
       "L 94.959937 221.481239 \n",
       "L 97.248385 222.784058 \n",
       "L 99.536834 223.875757 \n",
       "L 102.588098 225.062643 \n",
       "L 105.639363 226.003025 \n",
       "L 109.453443 226.910853 \n",
       "L 113.267524 227.59323 \n",
       "L 117.844421 228.194788 \n",
       "L 123.94695 228.741296 \n",
       "L 131.575111 229.163915 \n",
       "L 141.491721 229.469742 \n",
       "L 156.748043 229.687556 \n",
       "L 184.209424 229.816787 \n",
       "L 257.439773 229.87053 \n",
       "L 358.894318 229.874489 \n",
       "L 358.894318 229.874489 \n",
       "\" style=\"fill:none;stroke:#4c72b0;stroke-linecap:round;stroke-width:2;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 39.3125 239.758125 \n",
       "L 39.3125 22.318125 \n",
       "\" style=\"fill:none;stroke:#262626;stroke-linecap:square;stroke-linejoin:miter;stroke-width:1.25;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 374.1125 239.758125 \n",
       "L 374.1125 22.318125 \n",
       "\" style=\"fill:none;stroke:#262626;stroke-linecap:square;stroke-linejoin:miter;stroke-width:1.25;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 39.3125 239.758125 \n",
       "L 374.1125 239.758125 \n",
       "\" style=\"fill:none;stroke:#262626;stroke-linecap:square;stroke-linejoin:miter;stroke-width:1.25;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 39.3125 22.318125 \n",
       "L 374.1125 22.318125 \n",
       "\" style=\"fill:none;stroke:#262626;stroke-linecap:square;stroke-linejoin:miter;stroke-width:1.25;\"/>\n",
       "   </g>\n",
       "   <g id=\"text_20\">\n",
       "    <!-- Covariance Graph with appropriate learning rate -->\n",
       "    <defs>\n",
       "     <path d=\"M 2.984375 54.6875 \n",
       "L 12.5 54.6875 \n",
       "L 29.59375 8.796875 \n",
       "L 46.6875 54.6875 \n",
       "L 56.203125 54.6875 \n",
       "L 35.6875 0 \n",
       "L 23.484375 0 \n",
       "z\n",
       "\" id=\"DejaVuSans-118\"/>\n",
       "     <path d=\"M 48.78125 52.59375 \n",
       "L 48.78125 44.1875 \n",
       "Q 44.96875 46.296875 41.140625 47.34375 \n",
       "Q 37.3125 48.390625 33.40625 48.390625 \n",
       "Q 24.65625 48.390625 19.8125 42.84375 \n",
       "Q 14.984375 37.3125 14.984375 27.296875 \n",
       "Q 14.984375 17.28125 19.8125 11.734375 \n",
       "Q 24.65625 6.203125 33.40625 6.203125 \n",
       "Q 37.3125 6.203125 41.140625 7.25 \n",
       "Q 44.96875 8.296875 48.78125 10.40625 \n",
       "L 48.78125 2.09375 \n",
       "Q 45.015625 0.34375 40.984375 -0.53125 \n",
       "Q 36.96875 -1.421875 32.421875 -1.421875 \n",
       "Q 20.0625 -1.421875 12.78125 6.34375 \n",
       "Q 5.515625 14.109375 5.515625 27.296875 \n",
       "Q 5.515625 40.671875 12.859375 48.328125 \n",
       "Q 20.21875 56 33.015625 56 \n",
       "Q 37.15625 56 41.109375 55.140625 \n",
       "Q 45.0625 54.296875 48.78125 52.59375 \n",
       "z\n",
       "\" id=\"DejaVuSans-99\"/>\n",
       "     <path d=\"M 59.515625 10.40625 \n",
       "L 59.515625 29.984375 \n",
       "L 43.40625 29.984375 \n",
       "L 43.40625 38.09375 \n",
       "L 69.28125 38.09375 \n",
       "L 69.28125 6.78125 \n",
       "Q 63.578125 2.734375 56.6875 0.65625 \n",
       "Q 49.8125 -1.421875 42 -1.421875 \n",
       "Q 24.90625 -1.421875 15.25 8.5625 \n",
       "Q 5.609375 18.5625 5.609375 36.375 \n",
       "Q 5.609375 54.25 15.25 64.234375 \n",
       "Q 24.90625 74.21875 42 74.21875 \n",
       "Q 49.125 74.21875 55.546875 72.453125 \n",
       "Q 61.96875 70.703125 67.390625 67.28125 \n",
       "L 67.390625 56.78125 \n",
       "Q 61.921875 61.421875 55.765625 63.765625 \n",
       "Q 49.609375 66.109375 42.828125 66.109375 \n",
       "Q 29.4375 66.109375 22.71875 58.640625 \n",
       "Q 16.015625 51.171875 16.015625 36.375 \n",
       "Q 16.015625 21.625 22.71875 14.15625 \n",
       "Q 29.4375 6.6875 42.828125 6.6875 \n",
       "Q 48.046875 6.6875 52.140625 7.59375 \n",
       "Q 56.25 8.5 59.515625 10.40625 \n",
       "z\n",
       "\" id=\"DejaVuSans-71\"/>\n",
       "     <path d=\"M 18.109375 8.203125 \n",
       "L 18.109375 -20.796875 \n",
       "L 9.078125 -20.796875 \n",
       "L 9.078125 54.6875 \n",
       "L 18.109375 54.6875 \n",
       "L 18.109375 46.390625 \n",
       "Q 20.953125 51.265625 25.265625 53.625 \n",
       "Q 29.59375 56 35.59375 56 \n",
       "Q 45.5625 56 51.78125 48.09375 \n",
       "Q 58.015625 40.1875 58.015625 27.296875 \n",
       "Q 58.015625 14.40625 51.78125 6.484375 \n",
       "Q 45.5625 -1.421875 35.59375 -1.421875 \n",
       "Q 29.59375 -1.421875 25.265625 0.953125 \n",
       "Q 20.953125 3.328125 18.109375 8.203125 \n",
       "z\n",
       "M 48.6875 27.296875 \n",
       "Q 48.6875 37.203125 44.609375 42.84375 \n",
       "Q 40.53125 48.484375 33.40625 48.484375 \n",
       "Q 26.265625 48.484375 22.1875 42.84375 \n",
       "Q 18.109375 37.203125 18.109375 27.296875 \n",
       "Q 18.109375 17.390625 22.1875 11.75 \n",
       "Q 26.265625 6.109375 33.40625 6.109375 \n",
       "Q 40.53125 6.109375 44.609375 11.75 \n",
       "Q 48.6875 17.390625 48.6875 27.296875 \n",
       "z\n",
       "\" id=\"DejaVuSans-112\"/>\n",
       "     <path d=\"M 54.890625 33.015625 \n",
       "L 54.890625 0 \n",
       "L 45.90625 0 \n",
       "L 45.90625 32.71875 \n",
       "Q 45.90625 40.484375 42.875 44.328125 \n",
       "Q 39.84375 48.1875 33.796875 48.1875 \n",
       "Q 26.515625 48.1875 22.3125 43.546875 \n",
       "Q 18.109375 38.921875 18.109375 30.90625 \n",
       "L 18.109375 0 \n",
       "L 9.078125 0 \n",
       "L 9.078125 75.984375 \n",
       "L 18.109375 75.984375 \n",
       "L 18.109375 46.1875 \n",
       "Q 21.34375 51.125 25.703125 53.5625 \n",
       "Q 30.078125 56 35.796875 56 \n",
       "Q 45.21875 56 50.046875 50.171875 \n",
       "Q 54.890625 44.34375 54.890625 33.015625 \n",
       "z\n",
       "\" id=\"DejaVuSans-104\"/>\n",
       "     <path d=\"M 4.203125 54.6875 \n",
       "L 13.1875 54.6875 \n",
       "L 24.421875 12.015625 \n",
       "L 35.59375 54.6875 \n",
       "L 46.1875 54.6875 \n",
       "L 57.421875 12.015625 \n",
       "L 68.609375 54.6875 \n",
       "L 77.59375 54.6875 \n",
       "L 63.28125 0 \n",
       "L 52.6875 0 \n",
       "L 40.921875 44.828125 \n",
       "L 29.109375 0 \n",
       "L 18.5 0 \n",
       "z\n",
       "\" id=\"DejaVuSans-119\"/>\n",
       "     <path d=\"M 9.421875 75.984375 \n",
       "L 18.40625 75.984375 \n",
       "L 18.40625 0 \n",
       "L 9.421875 0 \n",
       "z\n",
       "\" id=\"DejaVuSans-108\"/>\n",
       "     <path d=\"M 45.40625 27.984375 \n",
       "Q 45.40625 37.75 41.375 43.109375 \n",
       "Q 37.359375 48.484375 30.078125 48.484375 \n",
       "Q 22.859375 48.484375 18.828125 43.109375 \n",
       "Q 14.796875 37.75 14.796875 27.984375 \n",
       "Q 14.796875 18.265625 18.828125 12.890625 \n",
       "Q 22.859375 7.515625 30.078125 7.515625 \n",
       "Q 37.359375 7.515625 41.375 12.890625 \n",
       "Q 45.40625 18.265625 45.40625 27.984375 \n",
       "z\n",
       "M 54.390625 6.78125 \n",
       "Q 54.390625 -7.171875 48.1875 -13.984375 \n",
       "Q 42 -20.796875 29.203125 -20.796875 \n",
       "Q 24.46875 -20.796875 20.265625 -20.09375 \n",
       "Q 16.0625 -19.390625 12.109375 -17.921875 \n",
       "L 12.109375 -9.1875 \n",
       "Q 16.0625 -11.328125 19.921875 -12.34375 \n",
       "Q 23.78125 -13.375 27.78125 -13.375 \n",
       "Q 36.625 -13.375 41.015625 -8.765625 \n",
       "Q 45.40625 -4.15625 45.40625 5.171875 \n",
       "L 45.40625 9.625 \n",
       "Q 42.625 4.78125 38.28125 2.390625 \n",
       "Q 33.9375 0 27.875 0 \n",
       "Q 17.828125 0 11.671875 7.65625 \n",
       "Q 5.515625 15.328125 5.515625 27.984375 \n",
       "Q 5.515625 40.671875 11.671875 48.328125 \n",
       "Q 17.828125 56 27.875 56 \n",
       "Q 33.9375 56 38.28125 53.609375 \n",
       "Q 42.625 51.21875 45.40625 46.390625 \n",
       "L 45.40625 54.6875 \n",
       "L 54.390625 54.6875 \n",
       "z\n",
       "\" id=\"DejaVuSans-103\"/>\n",
       "    </defs>\n",
       "    <g style=\"fill:#262626;\" transform=\"translate(60.48875 16.318125)scale(0.12 -0.12)\">\n",
       "     <use xlink:href=\"#DejaVuSans-67\"/>\n",
       "     <use x=\"69.824219\" xlink:href=\"#DejaVuSans-111\"/>\n",
       "     <use x=\"131.005859\" xlink:href=\"#DejaVuSans-118\"/>\n",
       "     <use x=\"190.185547\" xlink:href=\"#DejaVuSans-97\"/>\n",
       "     <use x=\"251.464844\" xlink:href=\"#DejaVuSans-114\"/>\n",
       "     <use x=\"292.578125\" xlink:href=\"#DejaVuSans-105\"/>\n",
       "     <use x=\"320.361328\" xlink:href=\"#DejaVuSans-97\"/>\n",
       "     <use x=\"381.640625\" xlink:href=\"#DejaVuSans-110\"/>\n",
       "     <use x=\"445.019531\" xlink:href=\"#DejaVuSans-99\"/>\n",
       "     <use x=\"500\" xlink:href=\"#DejaVuSans-101\"/>\n",
       "     <use x=\"561.523438\" xlink:href=\"#DejaVuSans-32\"/>\n",
       "     <use x=\"593.310547\" xlink:href=\"#DejaVuSans-71\"/>\n",
       "     <use x=\"670.800781\" xlink:href=\"#DejaVuSans-114\"/>\n",
       "     <use x=\"711.914062\" xlink:href=\"#DejaVuSans-97\"/>\n",
       "     <use x=\"773.193359\" xlink:href=\"#DejaVuSans-112\"/>\n",
       "     <use x=\"836.669922\" xlink:href=\"#DejaVuSans-104\"/>\n",
       "     <use x=\"900.048828\" xlink:href=\"#DejaVuSans-32\"/>\n",
       "     <use x=\"931.835938\" xlink:href=\"#DejaVuSans-119\"/>\n",
       "     <use x=\"1013.623047\" xlink:href=\"#DejaVuSans-105\"/>\n",
       "     <use x=\"1041.40625\" xlink:href=\"#DejaVuSans-116\"/>\n",
       "     <use x=\"1080.615234\" xlink:href=\"#DejaVuSans-104\"/>\n",
       "     <use x=\"1143.994141\" xlink:href=\"#DejaVuSans-32\"/>\n",
       "     <use x=\"1175.78125\" xlink:href=\"#DejaVuSans-97\"/>\n",
       "     <use x=\"1237.060547\" xlink:href=\"#DejaVuSans-112\"/>\n",
       "     <use x=\"1300.537109\" xlink:href=\"#DejaVuSans-112\"/>\n",
       "     <use x=\"1364.013672\" xlink:href=\"#DejaVuSans-114\"/>\n",
       "     <use x=\"1405.095703\" xlink:href=\"#DejaVuSans-111\"/>\n",
       "     <use x=\"1466.277344\" xlink:href=\"#DejaVuSans-112\"/>\n",
       "     <use x=\"1529.753906\" xlink:href=\"#DejaVuSans-114\"/>\n",
       "     <use x=\"1570.867188\" xlink:href=\"#DejaVuSans-105\"/>\n",
       "     <use x=\"1598.650391\" xlink:href=\"#DejaVuSans-97\"/>\n",
       "     <use x=\"1659.929688\" xlink:href=\"#DejaVuSans-116\"/>\n",
       "     <use x=\"1699.138672\" xlink:href=\"#DejaVuSans-101\"/>\n",
       "     <use x=\"1760.662109\" xlink:href=\"#DejaVuSans-32\"/>\n",
       "     <use x=\"1792.449219\" xlink:href=\"#DejaVuSans-108\"/>\n",
       "     <use x=\"1820.232422\" xlink:href=\"#DejaVuSans-101\"/>\n",
       "     <use x=\"1881.755859\" xlink:href=\"#DejaVuSans-97\"/>\n",
       "     <use x=\"1943.035156\" xlink:href=\"#DejaVuSans-114\"/>\n",
       "     <use x=\"1984.132812\" xlink:href=\"#DejaVuSans-110\"/>\n",
       "     <use x=\"2047.511719\" xlink:href=\"#DejaVuSans-105\"/>\n",
       "     <use x=\"2075.294922\" xlink:href=\"#DejaVuSans-110\"/>\n",
       "     <use x=\"2138.673828\" xlink:href=\"#DejaVuSans-103\"/>\n",
       "     <use x=\"2202.150391\" xlink:href=\"#DejaVuSans-32\"/>\n",
       "     <use x=\"2233.9375\" xlink:href=\"#DejaVuSans-114\"/>\n",
       "     <use x=\"2275.050781\" xlink:href=\"#DejaVuSans-97\"/>\n",
       "     <use x=\"2336.330078\" xlink:href=\"#DejaVuSans-116\"/>\n",
       "     <use x=\"2375.539062\" xlink:href=\"#DejaVuSans-101\"/>\n",
       "    </g>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"p881de1886d\">\n",
       "   <rect height=\"217.44\" width=\"334.8\" x=\"39.3125\" y=\"22.318125\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "print('Running gradient descent ...')\n",
    "\n",
    "# Choose some alpha value\n",
    "alpha = 0.03\n",
    "num_iters = 400\n",
    "\n",
    "# Init Theta and Run Gradient Descent\n",
    "initial_theta = np.zeros((X.shape[1], 1))\n",
    "theta, J_history = gradient_descent(X, y, initial_theta, alpha, num_iters)\n",
    "\n",
    "print(\"Obtained theta is: {}\".format(theta.ravel()))\n",
    "\n",
    "# Covariance Graph with appropriate learning rate\n",
    "plt.plot(range(J_history.size), J_history, lw=2)\n",
    "plt.xlabel(\"Number of Iterations\")\n",
    "plt.ylabel(\"Cost J\")\n",
    "plt.title(\"Covariance Graph with appropriate learning rate\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Selecting Learning Rates\n",
    "\n",
    "In this part of the exercise, you will get to try out different learning rates for\n",
    "the dataset and find a learning rate that converges quickly. \n",
    "\n",
    "The next phase we are going to call our gradient_descent function and run gradient descent for about 50 iterations at the chosen learning rate. The function should also return the history of J(θ) values in a vector J_history. After the last iteration, we are going to plot the J values against the number of the iterations.\n",
    "\n",
    "Note: If your graph looks very different, especially if your value of J(θ)\n",
    "increases or even blows up, adjust your learning rate and try again. We recommend trying values of the learning rate α on a log-scale, at multiplicative\n",
    "steps of about 3 times the previous value (i.e., 0.3, 0.1, 0.03, 0.01 and so on).\n",
    "You may also want to adjust the number of iterations you are running if that\n",
    "will help you see the overall trend in the curve."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0, 0.5, 'Cost J')"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Created with matplotlib (https://matplotlib.org/) -->\n",
       "<svg height=\"282.717813pt\" version=\"1.1\" viewBox=\"0 0 381.3125 282.717813\" width=\"381.3125pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       " <defs>\n",
       "  <style type=\"text/css\">\n",
       "*{stroke-linecap:butt;stroke-linejoin:round;}\n",
       "  </style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 282.717813 \n",
       "L 381.3125 282.717813 \n",
       "L 381.3125 0 \n",
       "L 0 0 \n",
       "z\n",
       "\" style=\"fill:none;\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 39.3125 239.758125 \n",
       "L 374.1125 239.758125 \n",
       "L 374.1125 22.318125 \n",
       "L 39.3125 22.318125 \n",
       "z\n",
       "\" style=\"fill:#ffffff;\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"text_1\">\n",
       "      <!-- 0 -->\n",
       "      <defs>\n",
       "       <path d=\"M 31.78125 66.40625 \n",
       "Q 24.171875 66.40625 20.328125 58.90625 \n",
       "Q 16.5 51.421875 16.5 36.375 \n",
       "Q 16.5 21.390625 20.328125 13.890625 \n",
       "Q 24.171875 6.390625 31.78125 6.390625 \n",
       "Q 39.453125 6.390625 43.28125 13.890625 \n",
       "Q 47.125 21.390625 47.125 36.375 \n",
       "Q 47.125 51.421875 43.28125 58.90625 \n",
       "Q 39.453125 66.40625 31.78125 66.40625 \n",
       "z\n",
       "M 31.78125 74.21875 \n",
       "Q 44.046875 74.21875 50.515625 64.515625 \n",
       "Q 56.984375 54.828125 56.984375 36.375 \n",
       "Q 56.984375 17.96875 50.515625 8.265625 \n",
       "Q 44.046875 -1.421875 31.78125 -1.421875 \n",
       "Q 19.53125 -1.421875 13.0625 8.265625 \n",
       "Q 6.59375 17.96875 6.59375 36.375 \n",
       "Q 6.59375 54.828125 13.0625 64.515625 \n",
       "Q 19.53125 74.21875 31.78125 74.21875 \n",
       "z\n",
       "\" id=\"DejaVuSans-48\"/>\n",
       "      </defs>\n",
       "      <g style=\"fill:#262626;\" transform=\"translate(51.031307 257.616406)scale(0.11 -0.11)\">\n",
       "       <use xlink:href=\"#DejaVuSans-48\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"text_2\">\n",
       "      <!-- 50 -->\n",
       "      <defs>\n",
       "       <path d=\"M 10.796875 72.90625 \n",
       "L 49.515625 72.90625 \n",
       "L 49.515625 64.59375 \n",
       "L 19.828125 64.59375 \n",
       "L 19.828125 46.734375 \n",
       "Q 21.96875 47.46875 24.109375 47.828125 \n",
       "Q 26.265625 48.1875 28.421875 48.1875 \n",
       "Q 40.625 48.1875 47.75 41.5 \n",
       "Q 54.890625 34.8125 54.890625 23.390625 \n",
       "Q 54.890625 11.625 47.5625 5.09375 \n",
       "Q 40.234375 -1.421875 26.90625 -1.421875 \n",
       "Q 22.3125 -1.421875 17.546875 -0.640625 \n",
       "Q 12.796875 0.140625 7.71875 1.703125 \n",
       "L 7.71875 11.625 \n",
       "Q 12.109375 9.234375 16.796875 8.0625 \n",
       "Q 21.484375 6.890625 26.703125 6.890625 \n",
       "Q 35.15625 6.890625 40.078125 11.328125 \n",
       "Q 45.015625 15.765625 45.015625 23.390625 \n",
       "Q 45.015625 31 40.078125 35.4375 \n",
       "Q 35.15625 39.890625 26.703125 39.890625 \n",
       "Q 22.75 39.890625 18.8125 39.015625 \n",
       "Q 14.890625 38.140625 10.796875 36.28125 \n",
       "z\n",
       "\" id=\"DejaVuSans-53\"/>\n",
       "      </defs>\n",
       "      <g style=\"fill:#262626;\" transform=\"translate(85.672738 257.616406)scale(0.11 -0.11)\">\n",
       "       <use xlink:href=\"#DejaVuSans-53\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 100 -->\n",
       "      <defs>\n",
       "       <path d=\"M 12.40625 8.296875 \n",
       "L 28.515625 8.296875 \n",
       "L 28.515625 63.921875 \n",
       "L 10.984375 60.40625 \n",
       "L 10.984375 69.390625 \n",
       "L 28.421875 72.90625 \n",
       "L 38.28125 72.90625 \n",
       "L 38.28125 8.296875 \n",
       "L 54.390625 8.296875 \n",
       "L 54.390625 0 \n",
       "L 12.40625 0 \n",
       "z\n",
       "\" id=\"DejaVuSans-49\"/>\n",
       "      </defs>\n",
       "      <g style=\"fill:#262626;\" transform=\"translate(120.31417 257.616406)scale(0.11 -0.11)\">\n",
       "       <use xlink:href=\"#DejaVuSans-49\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\n",
       "       <use x=\"127.246094\" xlink:href=\"#DejaVuSans-48\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"text_4\">\n",
       "      <!-- 150 -->\n",
       "      <g style=\"fill:#262626;\" transform=\"translate(158.454977 257.616406)scale(0.11 -0.11)\">\n",
       "       <use xlink:href=\"#DejaVuSans-49\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-53\"/>\n",
       "       <use x=\"127.246094\" xlink:href=\"#DejaVuSans-48\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"text_5\">\n",
       "      <!-- 200 -->\n",
       "      <defs>\n",
       "       <path d=\"M 19.1875 8.296875 \n",
       "L 53.609375 8.296875 \n",
       "L 53.609375 0 \n",
       "L 7.328125 0 \n",
       "L 7.328125 8.296875 \n",
       "Q 12.9375 14.109375 22.625 23.890625 \n",
       "Q 32.328125 33.6875 34.8125 36.53125 \n",
       "Q 39.546875 41.84375 41.421875 45.53125 \n",
       "Q 43.3125 49.21875 43.3125 52.78125 \n",
       "Q 43.3125 58.59375 39.234375 62.25 \n",
       "Q 35.15625 65.921875 28.609375 65.921875 \n",
       "Q 23.96875 65.921875 18.8125 64.3125 \n",
       "Q 13.671875 62.703125 7.8125 59.421875 \n",
       "L 7.8125 69.390625 \n",
       "Q 13.765625 71.78125 18.9375 73 \n",
       "Q 24.125 74.21875 28.421875 74.21875 \n",
       "Q 39.75 74.21875 46.484375 68.546875 \n",
       "Q 53.21875 62.890625 53.21875 53.421875 \n",
       "Q 53.21875 48.921875 51.53125 44.890625 \n",
       "Q 49.859375 40.875 45.40625 35.40625 \n",
       "Q 44.1875 33.984375 37.640625 27.21875 \n",
       "Q 31.109375 20.453125 19.1875 8.296875 \n",
       "z\n",
       "\" id=\"DejaVuSans-50\"/>\n",
       "      </defs>\n",
       "      <g style=\"fill:#262626;\" transform=\"translate(196.595783 257.616406)scale(0.11 -0.11)\">\n",
       "       <use xlink:href=\"#DejaVuSans-50\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\n",
       "       <use x=\"127.246094\" xlink:href=\"#DejaVuSans-48\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_6\">\n",
       "     <g id=\"text_6\">\n",
       "      <!-- 250 -->\n",
       "      <g style=\"fill:#262626;\" transform=\"translate(234.73659 257.616406)scale(0.11 -0.11)\">\n",
       "       <use xlink:href=\"#DejaVuSans-50\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-53\"/>\n",
       "       <use x=\"127.246094\" xlink:href=\"#DejaVuSans-48\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_7\">\n",
       "     <g id=\"text_7\">\n",
       "      <!-- 300 -->\n",
       "      <defs>\n",
       "       <path d=\"M 40.578125 39.3125 \n",
       "Q 47.65625 37.796875 51.625 33 \n",
       "Q 55.609375 28.21875 55.609375 21.1875 \n",
       "Q 55.609375 10.40625 48.1875 4.484375 \n",
       "Q 40.765625 -1.421875 27.09375 -1.421875 \n",
       "Q 22.515625 -1.421875 17.65625 -0.515625 \n",
       "Q 12.796875 0.390625 7.625 2.203125 \n",
       "L 7.625 11.71875 \n",
       "Q 11.71875 9.328125 16.59375 8.109375 \n",
       "Q 21.484375 6.890625 26.8125 6.890625 \n",
       "Q 36.078125 6.890625 40.9375 10.546875 \n",
       "Q 45.796875 14.203125 45.796875 21.1875 \n",
       "Q 45.796875 27.640625 41.28125 31.265625 \n",
       "Q 36.765625 34.90625 28.71875 34.90625 \n",
       "L 20.21875 34.90625 \n",
       "L 20.21875 43.015625 \n",
       "L 29.109375 43.015625 \n",
       "Q 36.375 43.015625 40.234375 45.921875 \n",
       "Q 44.09375 48.828125 44.09375 54.296875 \n",
       "Q 44.09375 59.90625 40.109375 62.90625 \n",
       "Q 36.140625 65.921875 28.71875 65.921875 \n",
       "Q 24.65625 65.921875 20.015625 65.03125 \n",
       "Q 15.375 64.15625 9.8125 62.3125 \n",
       "L 9.8125 71.09375 \n",
       "Q 15.4375 72.65625 20.34375 73.4375 \n",
       "Q 25.25 74.21875 29.59375 74.21875 \n",
       "Q 40.828125 74.21875 47.359375 69.109375 \n",
       "Q 53.90625 64.015625 53.90625 55.328125 \n",
       "Q 53.90625 49.265625 50.4375 45.09375 \n",
       "Q 46.96875 40.921875 40.578125 39.3125 \n",
       "z\n",
       "\" id=\"DejaVuSans-51\"/>\n",
       "      </defs>\n",
       "      <g style=\"fill:#262626;\" transform=\"translate(272.877396 257.616406)scale(0.11 -0.11)\">\n",
       "       <use xlink:href=\"#DejaVuSans-51\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\n",
       "       <use x=\"127.246094\" xlink:href=\"#DejaVuSans-48\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_8\">\n",
       "     <g id=\"text_8\">\n",
       "      <!-- 350 -->\n",
       "      <g style=\"fill:#262626;\" transform=\"translate(311.018203 257.616406)scale(0.11 -0.11)\">\n",
       "       <use xlink:href=\"#DejaVuSans-51\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-53\"/>\n",
       "       <use x=\"127.246094\" xlink:href=\"#DejaVuSans-48\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_9\">\n",
       "     <g id=\"text_9\">\n",
       "      <!-- 400 -->\n",
       "      <defs>\n",
       "       <path d=\"M 37.796875 64.3125 \n",
       "L 12.890625 25.390625 \n",
       "L 37.796875 25.390625 \n",
       "z\n",
       "M 35.203125 72.90625 \n",
       "L 47.609375 72.90625 \n",
       "L 47.609375 25.390625 \n",
       "L 58.015625 25.390625 \n",
       "L 58.015625 17.1875 \n",
       "L 47.609375 17.1875 \n",
       "L 47.609375 0 \n",
       "L 37.796875 0 \n",
       "L 37.796875 17.1875 \n",
       "L 4.890625 17.1875 \n",
       "L 4.890625 26.703125 \n",
       "z\n",
       "\" id=\"DejaVuSans-52\"/>\n",
       "      </defs>\n",
       "      <g style=\"fill:#262626;\" transform=\"translate(349.159009 257.616406)scale(0.11 -0.11)\">\n",
       "       <use xlink:href=\"#DejaVuSans-52\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\n",
       "       <use x=\"127.246094\" xlink:href=\"#DejaVuSans-48\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_10\">\n",
       "     <!-- Number of iterations -->\n",
       "     <defs>\n",
       "      <path d=\"M 9.8125 72.90625 \n",
       "L 23.09375 72.90625 \n",
       "L 55.421875 11.921875 \n",
       "L 55.421875 72.90625 \n",
       "L 64.984375 72.90625 \n",
       "L 64.984375 0 \n",
       "L 51.703125 0 \n",
       "L 19.390625 60.984375 \n",
       "L 19.390625 0 \n",
       "L 9.8125 0 \n",
       "z\n",
       "\" id=\"DejaVuSans-78\"/>\n",
       "      <path d=\"M 8.5 21.578125 \n",
       "L 8.5 54.6875 \n",
       "L 17.484375 54.6875 \n",
       "L 17.484375 21.921875 \n",
       "Q 17.484375 14.15625 20.5 10.265625 \n",
       "Q 23.53125 6.390625 29.59375 6.390625 \n",
       "Q 36.859375 6.390625 41.078125 11.03125 \n",
       "Q 45.3125 15.671875 45.3125 23.6875 \n",
       "L 45.3125 54.6875 \n",
       "L 54.296875 54.6875 \n",
       "L 54.296875 0 \n",
       "L 45.3125 0 \n",
       "L 45.3125 8.40625 \n",
       "Q 42.046875 3.421875 37.71875 1 \n",
       "Q 33.40625 -1.421875 27.6875 -1.421875 \n",
       "Q 18.265625 -1.421875 13.375 4.4375 \n",
       "Q 8.5 10.296875 8.5 21.578125 \n",
       "z\n",
       "M 31.109375 56 \n",
       "z\n",
       "\" id=\"DejaVuSans-117\"/>\n",
       "      <path d=\"M 52 44.1875 \n",
       "Q 55.375 50.25 60.0625 53.125 \n",
       "Q 64.75 56 71.09375 56 \n",
       "Q 79.640625 56 84.28125 50.015625 \n",
       "Q 88.921875 44.046875 88.921875 33.015625 \n",
       "L 88.921875 0 \n",
       "L 79.890625 0 \n",
       "L 79.890625 32.71875 \n",
       "Q 79.890625 40.578125 77.09375 44.375 \n",
       "Q 74.3125 48.1875 68.609375 48.1875 \n",
       "Q 61.625 48.1875 57.5625 43.546875 \n",
       "Q 53.515625 38.921875 53.515625 30.90625 \n",
       "L 53.515625 0 \n",
       "L 44.484375 0 \n",
       "L 44.484375 32.71875 \n",
       "Q 44.484375 40.625 41.703125 44.40625 \n",
       "Q 38.921875 48.1875 33.109375 48.1875 \n",
       "Q 26.21875 48.1875 22.15625 43.53125 \n",
       "Q 18.109375 38.875 18.109375 30.90625 \n",
       "L 18.109375 0 \n",
       "L 9.078125 0 \n",
       "L 9.078125 54.6875 \n",
       "L 18.109375 54.6875 \n",
       "L 18.109375 46.1875 \n",
       "Q 21.1875 51.21875 25.484375 53.609375 \n",
       "Q 29.78125 56 35.6875 56 \n",
       "Q 41.65625 56 45.828125 52.96875 \n",
       "Q 50 49.953125 52 44.1875 \n",
       "z\n",
       "\" id=\"DejaVuSans-109\"/>\n",
       "      <path d=\"M 48.6875 27.296875 \n",
       "Q 48.6875 37.203125 44.609375 42.84375 \n",
       "Q 40.53125 48.484375 33.40625 48.484375 \n",
       "Q 26.265625 48.484375 22.1875 42.84375 \n",
       "Q 18.109375 37.203125 18.109375 27.296875 \n",
       "Q 18.109375 17.390625 22.1875 11.75 \n",
       "Q 26.265625 6.109375 33.40625 6.109375 \n",
       "Q 40.53125 6.109375 44.609375 11.75 \n",
       "Q 48.6875 17.390625 48.6875 27.296875 \n",
       "z\n",
       "M 18.109375 46.390625 \n",
       "Q 20.953125 51.265625 25.265625 53.625 \n",
       "Q 29.59375 56 35.59375 56 \n",
       "Q 45.5625 56 51.78125 48.09375 \n",
       "Q 58.015625 40.1875 58.015625 27.296875 \n",
       "Q 58.015625 14.40625 51.78125 6.484375 \n",
       "Q 45.5625 -1.421875 35.59375 -1.421875 \n",
       "Q 29.59375 -1.421875 25.265625 0.953125 \n",
       "Q 20.953125 3.328125 18.109375 8.203125 \n",
       "L 18.109375 0 \n",
       "L 9.078125 0 \n",
       "L 9.078125 75.984375 \n",
       "L 18.109375 75.984375 \n",
       "z\n",
       "\" id=\"DejaVuSans-98\"/>\n",
       "      <path d=\"M 56.203125 29.59375 \n",
       "L 56.203125 25.203125 \n",
       "L 14.890625 25.203125 \n",
       "Q 15.484375 15.921875 20.484375 11.0625 \n",
       "Q 25.484375 6.203125 34.421875 6.203125 \n",
       "Q 39.59375 6.203125 44.453125 7.46875 \n",
       "Q 49.3125 8.734375 54.109375 11.28125 \n",
       "L 54.109375 2.78125 \n",
       "Q 49.265625 0.734375 44.1875 -0.34375 \n",
       "Q 39.109375 -1.421875 33.890625 -1.421875 \n",
       "Q 20.796875 -1.421875 13.15625 6.1875 \n",
       "Q 5.515625 13.8125 5.515625 26.8125 \n",
       "Q 5.515625 40.234375 12.765625 48.109375 \n",
       "Q 20.015625 56 32.328125 56 \n",
       "Q 43.359375 56 49.78125 48.890625 \n",
       "Q 56.203125 41.796875 56.203125 29.59375 \n",
       "z\n",
       "M 47.21875 32.234375 \n",
       "Q 47.125 39.59375 43.09375 43.984375 \n",
       "Q 39.0625 48.390625 32.421875 48.390625 \n",
       "Q 24.90625 48.390625 20.390625 44.140625 \n",
       "Q 15.875 39.890625 15.1875 32.171875 \n",
       "z\n",
       "\" id=\"DejaVuSans-101\"/>\n",
       "      <path d=\"M 41.109375 46.296875 \n",
       "Q 39.59375 47.171875 37.8125 47.578125 \n",
       "Q 36.03125 48 33.890625 48 \n",
       "Q 26.265625 48 22.1875 43.046875 \n",
       "Q 18.109375 38.09375 18.109375 28.8125 \n",
       "L 18.109375 0 \n",
       "L 9.078125 0 \n",
       "L 9.078125 54.6875 \n",
       "L 18.109375 54.6875 \n",
       "L 18.109375 46.1875 \n",
       "Q 20.953125 51.171875 25.484375 53.578125 \n",
       "Q 30.03125 56 36.53125 56 \n",
       "Q 37.453125 56 38.578125 55.875 \n",
       "Q 39.703125 55.765625 41.0625 55.515625 \n",
       "z\n",
       "\" id=\"DejaVuSans-114\"/>\n",
       "      <path id=\"DejaVuSans-32\"/>\n",
       "      <path d=\"M 30.609375 48.390625 \n",
       "Q 23.390625 48.390625 19.1875 42.75 \n",
       "Q 14.984375 37.109375 14.984375 27.296875 \n",
       "Q 14.984375 17.484375 19.15625 11.84375 \n",
       "Q 23.34375 6.203125 30.609375 6.203125 \n",
       "Q 37.796875 6.203125 41.984375 11.859375 \n",
       "Q 46.1875 17.53125 46.1875 27.296875 \n",
       "Q 46.1875 37.015625 41.984375 42.703125 \n",
       "Q 37.796875 48.390625 30.609375 48.390625 \n",
       "z\n",
       "M 30.609375 56 \n",
       "Q 42.328125 56 49.015625 48.375 \n",
       "Q 55.71875 40.765625 55.71875 27.296875 \n",
       "Q 55.71875 13.875 49.015625 6.21875 \n",
       "Q 42.328125 -1.421875 30.609375 -1.421875 \n",
       "Q 18.84375 -1.421875 12.171875 6.21875 \n",
       "Q 5.515625 13.875 5.515625 27.296875 \n",
       "Q 5.515625 40.765625 12.171875 48.375 \n",
       "Q 18.84375 56 30.609375 56 \n",
       "z\n",
       "\" id=\"DejaVuSans-111\"/>\n",
       "      <path d=\"M 37.109375 75.984375 \n",
       "L 37.109375 68.5 \n",
       "L 28.515625 68.5 \n",
       "Q 23.6875 68.5 21.796875 66.546875 \n",
       "Q 19.921875 64.59375 19.921875 59.515625 \n",
       "L 19.921875 54.6875 \n",
       "L 34.71875 54.6875 \n",
       "L 34.71875 47.703125 \n",
       "L 19.921875 47.703125 \n",
       "L 19.921875 0 \n",
       "L 10.890625 0 \n",
       "L 10.890625 47.703125 \n",
       "L 2.296875 47.703125 \n",
       "L 2.296875 54.6875 \n",
       "L 10.890625 54.6875 \n",
       "L 10.890625 58.5 \n",
       "Q 10.890625 67.625 15.140625 71.796875 \n",
       "Q 19.390625 75.984375 28.609375 75.984375 \n",
       "z\n",
       "\" id=\"DejaVuSans-102\"/>\n",
       "      <path d=\"M 9.421875 54.6875 \n",
       "L 18.40625 54.6875 \n",
       "L 18.40625 0 \n",
       "L 9.421875 0 \n",
       "z\n",
       "M 9.421875 75.984375 \n",
       "L 18.40625 75.984375 \n",
       "L 18.40625 64.59375 \n",
       "L 9.421875 64.59375 \n",
       "z\n",
       "\" id=\"DejaVuSans-105\"/>\n",
       "      <path d=\"M 18.3125 70.21875 \n",
       "L 18.3125 54.6875 \n",
       "L 36.8125 54.6875 \n",
       "L 36.8125 47.703125 \n",
       "L 18.3125 47.703125 \n",
       "L 18.3125 18.015625 \n",
       "Q 18.3125 11.328125 20.140625 9.421875 \n",
       "Q 21.96875 7.515625 27.59375 7.515625 \n",
       "L 36.8125 7.515625 \n",
       "L 36.8125 0 \n",
       "L 27.59375 0 \n",
       "Q 17.1875 0 13.234375 3.875 \n",
       "Q 9.28125 7.765625 9.28125 18.015625 \n",
       "L 9.28125 47.703125 \n",
       "L 2.6875 47.703125 \n",
       "L 2.6875 54.6875 \n",
       "L 9.28125 54.6875 \n",
       "L 9.28125 70.21875 \n",
       "z\n",
       "\" id=\"DejaVuSans-116\"/>\n",
       "      <path d=\"M 34.28125 27.484375 \n",
       "Q 23.390625 27.484375 19.1875 25 \n",
       "Q 14.984375 22.515625 14.984375 16.5 \n",
       "Q 14.984375 11.71875 18.140625 8.90625 \n",
       "Q 21.296875 6.109375 26.703125 6.109375 \n",
       "Q 34.1875 6.109375 38.703125 11.40625 \n",
       "Q 43.21875 16.703125 43.21875 25.484375 \n",
       "L 43.21875 27.484375 \n",
       "z\n",
       "M 52.203125 31.203125 \n",
       "L 52.203125 0 \n",
       "L 43.21875 0 \n",
       "L 43.21875 8.296875 \n",
       "Q 40.140625 3.328125 35.546875 0.953125 \n",
       "Q 30.953125 -1.421875 24.3125 -1.421875 \n",
       "Q 15.921875 -1.421875 10.953125 3.296875 \n",
       "Q 6 8.015625 6 15.921875 \n",
       "Q 6 25.140625 12.171875 29.828125 \n",
       "Q 18.359375 34.515625 30.609375 34.515625 \n",
       "L 43.21875 34.515625 \n",
       "L 43.21875 35.40625 \n",
       "Q 43.21875 41.609375 39.140625 45 \n",
       "Q 35.0625 48.390625 27.6875 48.390625 \n",
       "Q 23 48.390625 18.546875 47.265625 \n",
       "Q 14.109375 46.140625 10.015625 43.890625 \n",
       "L 10.015625 52.203125 \n",
       "Q 14.9375 54.109375 19.578125 55.046875 \n",
       "Q 24.21875 56 28.609375 56 \n",
       "Q 40.484375 56 46.34375 49.84375 \n",
       "Q 52.203125 43.703125 52.203125 31.203125 \n",
       "z\n",
       "\" id=\"DejaVuSans-97\"/>\n",
       "      <path d=\"M 54.890625 33.015625 \n",
       "L 54.890625 0 \n",
       "L 45.90625 0 \n",
       "L 45.90625 32.71875 \n",
       "Q 45.90625 40.484375 42.875 44.328125 \n",
       "Q 39.84375 48.1875 33.796875 48.1875 \n",
       "Q 26.515625 48.1875 22.3125 43.546875 \n",
       "Q 18.109375 38.921875 18.109375 30.90625 \n",
       "L 18.109375 0 \n",
       "L 9.078125 0 \n",
       "L 9.078125 54.6875 \n",
       "L 18.109375 54.6875 \n",
       "L 18.109375 46.1875 \n",
       "Q 21.34375 51.125 25.703125 53.5625 \n",
       "Q 30.078125 56 35.796875 56 \n",
       "Q 45.21875 56 50.046875 50.171875 \n",
       "Q 54.890625 44.34375 54.890625 33.015625 \n",
       "z\n",
       "\" id=\"DejaVuSans-110\"/>\n",
       "      <path d=\"M 44.28125 53.078125 \n",
       "L 44.28125 44.578125 \n",
       "Q 40.484375 46.53125 36.375 47.5 \n",
       "Q 32.28125 48.484375 27.875 48.484375 \n",
       "Q 21.1875 48.484375 17.84375 46.4375 \n",
       "Q 14.5 44.390625 14.5 40.28125 \n",
       "Q 14.5 37.15625 16.890625 35.375 \n",
       "Q 19.28125 33.59375 26.515625 31.984375 \n",
       "L 29.59375 31.296875 \n",
       "Q 39.15625 29.25 43.1875 25.515625 \n",
       "Q 47.21875 21.78125 47.21875 15.09375 \n",
       "Q 47.21875 7.46875 41.1875 3.015625 \n",
       "Q 35.15625 -1.421875 24.609375 -1.421875 \n",
       "Q 20.21875 -1.421875 15.453125 -0.5625 \n",
       "Q 10.6875 0.296875 5.421875 2 \n",
       "L 5.421875 11.28125 \n",
       "Q 10.40625 8.6875 15.234375 7.390625 \n",
       "Q 20.0625 6.109375 24.8125 6.109375 \n",
       "Q 31.15625 6.109375 34.5625 8.28125 \n",
       "Q 37.984375 10.453125 37.984375 14.40625 \n",
       "Q 37.984375 18.0625 35.515625 20.015625 \n",
       "Q 33.0625 21.96875 24.703125 23.78125 \n",
       "L 21.578125 24.515625 \n",
       "Q 13.234375 26.265625 9.515625 29.90625 \n",
       "Q 5.8125 33.546875 5.8125 39.890625 \n",
       "Q 5.8125 47.609375 11.28125 51.796875 \n",
       "Q 16.75 56 26.8125 56 \n",
       "Q 31.78125 56 36.171875 55.265625 \n",
       "Q 40.578125 54.546875 44.28125 53.078125 \n",
       "z\n",
       "\" id=\"DejaVuSans-115\"/>\n",
       "     </defs>\n",
       "     <g style=\"fill:#262626;\" transform=\"translate(144.539375 273.022188)scale(0.12 -0.12)\">\n",
       "      <use xlink:href=\"#DejaVuSans-78\"/>\n",
       "      <use x=\"74.804688\" xlink:href=\"#DejaVuSans-117\"/>\n",
       "      <use x=\"138.183594\" xlink:href=\"#DejaVuSans-109\"/>\n",
       "      <use x=\"235.595703\" xlink:href=\"#DejaVuSans-98\"/>\n",
       "      <use x=\"299.072266\" xlink:href=\"#DejaVuSans-101\"/>\n",
       "      <use x=\"360.595703\" xlink:href=\"#DejaVuSans-114\"/>\n",
       "      <use x=\"401.708984\" xlink:href=\"#DejaVuSans-32\"/>\n",
       "      <use x=\"433.496094\" xlink:href=\"#DejaVuSans-111\"/>\n",
       "      <use x=\"494.677734\" xlink:href=\"#DejaVuSans-102\"/>\n",
       "      <use x=\"529.882812\" xlink:href=\"#DejaVuSans-32\"/>\n",
       "      <use x=\"561.669922\" xlink:href=\"#DejaVuSans-105\"/>\n",
       "      <use x=\"589.453125\" xlink:href=\"#DejaVuSans-116\"/>\n",
       "      <use x=\"628.662109\" xlink:href=\"#DejaVuSans-101\"/>\n",
       "      <use x=\"690.185547\" xlink:href=\"#DejaVuSans-114\"/>\n",
       "      <use x=\"731.298828\" xlink:href=\"#DejaVuSans-97\"/>\n",
       "      <use x=\"792.578125\" xlink:href=\"#DejaVuSans-116\"/>\n",
       "      <use x=\"831.787109\" xlink:href=\"#DejaVuSans-105\"/>\n",
       "      <use x=\"859.570312\" xlink:href=\"#DejaVuSans-111\"/>\n",
       "      <use x=\"920.751953\" xlink:href=\"#DejaVuSans-110\"/>\n",
       "      <use x=\"984.130859\" xlink:href=\"#DejaVuSans-115\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"text_11\">\n",
       "      <!-- 0 -->\n",
       "      <g style=\"fill:#262626;\" transform=\"translate(22.81375 240.448642)scale(0.11 -0.11)\">\n",
       "       <use xlink:href=\"#DejaVuSans-48\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"text_12\">\n",
       "      <!-- 1 -->\n",
       "      <g style=\"fill:#262626;\" transform=\"translate(22.81375 209.150861)scale(0.11 -0.11)\">\n",
       "       <use xlink:href=\"#DejaVuSans-49\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"text_13\">\n",
       "      <!-- 2 -->\n",
       "      <g style=\"fill:#262626;\" transform=\"translate(22.81375 177.85308)scale(0.11 -0.11)\">\n",
       "       <use xlink:href=\"#DejaVuSans-50\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_4\">\n",
       "     <g id=\"text_14\">\n",
       "      <!-- 3 -->\n",
       "      <g style=\"fill:#262626;\" transform=\"translate(22.81375 146.555298)scale(0.11 -0.11)\">\n",
       "       <use xlink:href=\"#DejaVuSans-51\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_5\">\n",
       "     <g id=\"text_15\">\n",
       "      <!-- 4 -->\n",
       "      <g style=\"fill:#262626;\" transform=\"translate(22.81375 115.257517)scale(0.11 -0.11)\">\n",
       "       <use xlink:href=\"#DejaVuSans-52\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_6\">\n",
       "     <g id=\"text_16\">\n",
       "      <!-- 5 -->\n",
       "      <g style=\"fill:#262626;\" transform=\"translate(22.81375 83.959736)scale(0.11 -0.11)\">\n",
       "       <use xlink:href=\"#DejaVuSans-53\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_7\">\n",
       "     <g id=\"text_17\">\n",
       "      <!-- 6 -->\n",
       "      <defs>\n",
       "       <path d=\"M 33.015625 40.375 \n",
       "Q 26.375 40.375 22.484375 35.828125 \n",
       "Q 18.609375 31.296875 18.609375 23.390625 \n",
       "Q 18.609375 15.53125 22.484375 10.953125 \n",
       "Q 26.375 6.390625 33.015625 6.390625 \n",
       "Q 39.65625 6.390625 43.53125 10.953125 \n",
       "Q 47.40625 15.53125 47.40625 23.390625 \n",
       "Q 47.40625 31.296875 43.53125 35.828125 \n",
       "Q 39.65625 40.375 33.015625 40.375 \n",
       "z\n",
       "M 52.59375 71.296875 \n",
       "L 52.59375 62.3125 \n",
       "Q 48.875 64.0625 45.09375 64.984375 \n",
       "Q 41.3125 65.921875 37.59375 65.921875 \n",
       "Q 27.828125 65.921875 22.671875 59.328125 \n",
       "Q 17.53125 52.734375 16.796875 39.40625 \n",
       "Q 19.671875 43.65625 24.015625 45.921875 \n",
       "Q 28.375 48.1875 33.59375 48.1875 \n",
       "Q 44.578125 48.1875 50.953125 41.515625 \n",
       "Q 57.328125 34.859375 57.328125 23.390625 \n",
       "Q 57.328125 12.15625 50.6875 5.359375 \n",
       "Q 44.046875 -1.421875 33.015625 -1.421875 \n",
       "Q 20.359375 -1.421875 13.671875 8.265625 \n",
       "Q 6.984375 17.96875 6.984375 36.375 \n",
       "Q 6.984375 53.65625 15.1875 63.9375 \n",
       "Q 23.390625 74.21875 37.203125 74.21875 \n",
       "Q 40.921875 74.21875 44.703125 73.484375 \n",
       "Q 48.484375 72.75 52.59375 71.296875 \n",
       "z\n",
       "\" id=\"DejaVuSans-54\"/>\n",
       "      </defs>\n",
       "      <g style=\"fill:#262626;\" transform=\"translate(22.81375 52.661954)scale(0.11 -0.11)\">\n",
       "       <use xlink:href=\"#DejaVuSans-54\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_18\">\n",
       "     <!-- Cost J -->\n",
       "     <defs>\n",
       "      <path d=\"M 64.40625 67.28125 \n",
       "L 64.40625 56.890625 \n",
       "Q 59.421875 61.53125 53.78125 63.8125 \n",
       "Q 48.140625 66.109375 41.796875 66.109375 \n",
       "Q 29.296875 66.109375 22.65625 58.46875 \n",
       "Q 16.015625 50.828125 16.015625 36.375 \n",
       "Q 16.015625 21.96875 22.65625 14.328125 \n",
       "Q 29.296875 6.6875 41.796875 6.6875 \n",
       "Q 48.140625 6.6875 53.78125 8.984375 \n",
       "Q 59.421875 11.28125 64.40625 15.921875 \n",
       "L 64.40625 5.609375 \n",
       "Q 59.234375 2.09375 53.4375 0.328125 \n",
       "Q 47.65625 -1.421875 41.21875 -1.421875 \n",
       "Q 24.65625 -1.421875 15.125 8.703125 \n",
       "Q 5.609375 18.84375 5.609375 36.375 \n",
       "Q 5.609375 53.953125 15.125 64.078125 \n",
       "Q 24.65625 74.21875 41.21875 74.21875 \n",
       "Q 47.75 74.21875 53.53125 72.484375 \n",
       "Q 59.328125 70.75 64.40625 67.28125 \n",
       "z\n",
       "\" id=\"DejaVuSans-67\"/>\n",
       "      <path d=\"M 9.8125 72.90625 \n",
       "L 19.671875 72.90625 \n",
       "L 19.671875 5.078125 \n",
       "Q 19.671875 -8.109375 14.671875 -14.0625 \n",
       "Q 9.671875 -20.015625 -1.421875 -20.015625 \n",
       "L -5.171875 -20.015625 \n",
       "L -5.171875 -11.71875 \n",
       "L -2.09375 -11.71875 \n",
       "Q 4.4375 -11.71875 7.125 -8.046875 \n",
       "Q 9.8125 -4.390625 9.8125 5.078125 \n",
       "z\n",
       "\" id=\"DejaVuSans-74\"/>\n",
       "     </defs>\n",
       "     <g style=\"fill:#262626;\" transform=\"translate(16.318125 148.05375)rotate(-90)scale(0.12 -0.12)\">\n",
       "      <use xlink:href=\"#DejaVuSans-67\"/>\n",
       "      <use x=\"69.824219\" xlink:href=\"#DejaVuSans-111\"/>\n",
       "      <use x=\"131.005859\" xlink:href=\"#DejaVuSans-115\"/>\n",
       "      <use x=\"183.105469\" xlink:href=\"#DejaVuSans-116\"/>\n",
       "      <use x=\"222.314453\" xlink:href=\"#DejaVuSans-32\"/>\n",
       "      <use x=\"254.101562\" xlink:href=\"#DejaVuSans-74\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_19\">\n",
       "     <!-- 1e10 -->\n",
       "     <g style=\"fill:#262626;\" transform=\"translate(39.3125 19.318125)scale(0.11 -0.11)\">\n",
       "      <use xlink:href=\"#DejaVuSans-49\"/>\n",
       "      <use x=\"63.623047\" xlink:href=\"#DejaVuSans-101\"/>\n",
       "      <use x=\"125.146484\" xlink:href=\"#DejaVuSans-49\"/>\n",
       "      <use x=\"188.769531\" xlink:href=\"#DejaVuSans-48\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"line2d_1\">\n",
       "    <path clip-path=\"url(#pd7e5f4e543)\" d=\"M 54.530682 32.201761 \n",
       "L 59.870395 40.524603 \n",
       "L 65.972924 49.597665 \n",
       "L 72.075453 58.225371 \n",
       "L 78.177982 66.43016 \n",
       "L 84.280511 74.233307 \n",
       "L 90.38304 81.654981 \n",
       "L 96.485569 88.714307 \n",
       "L 102.588098 95.429421 \n",
       "L 108.690627 101.817525 \n",
       "L 114.793156 107.894934 \n",
       "L 120.895685 113.677127 \n",
       "L 126.998214 119.178787 \n",
       "L 133.863559 125.050152 \n",
       "L 140.728905 130.602847 \n",
       "L 147.59425 135.854566 \n",
       "L 154.459595 140.821992 \n",
       "L 161.32494 145.52086 \n",
       "L 168.190285 149.966011 \n",
       "L 175.818447 154.624498 \n",
       "L 183.446608 159.005079 \n",
       "L 191.074769 163.124684 \n",
       "L 198.702931 166.999189 \n",
       "L 207.093908 170.995792 \n",
       "L 215.484886 174.73234 \n",
       "L 223.875863 178.226097 \n",
       "L 232.26684 181.493156 \n",
       "L 241.420634 184.816275 \n",
       "L 250.574428 187.905545 \n",
       "L 260.491037 191.007791 \n",
       "L 270.407647 193.874967 \n",
       "L 281.087073 196.720599 \n",
       "L 291.766499 199.335463 \n",
       "L 303.208741 201.902655 \n",
       "L 314.650983 204.248219 \n",
       "L 326.856041 206.527804 \n",
       "L 339.823915 208.721833 \n",
       "L 353.554605 210.814104 \n",
       "L 358.894318 211.568926 \n",
       "L 358.894318 211.568926 \n",
       "\" style=\"fill:none;stroke:#64b5cd;stroke-linecap:round;stroke-width:2;\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_2\">\n",
       "    <path clip-path=\"url(#pd7e5f4e543)\" d=\"M 54.530682 43.001963 \n",
       "L 56.056314 64.854759 \n",
       "L 57.581946 84.093248 \n",
       "L 59.107579 101.037892 \n",
       "L 60.633211 115.968709 \n",
       "L 62.158843 129.130562 \n",
       "L 63.684475 140.737723 \n",
       "L 65.210108 150.977823 \n",
       "L 66.73574 160.015274 \n",
       "L 68.261372 167.994232 \n",
       "L 69.787004 175.041166 \n",
       "L 71.312637 181.267093 \n",
       "L 72.838269 186.769517 \n",
       "L 74.363901 191.63412 \n",
       "L 75.889533 195.936231 \n",
       "L 77.415166 199.742107 \n",
       "L 78.940798 203.110056 \n",
       "L 80.46643 206.091405 \n",
       "L 81.992063 208.731361 \n",
       "L 83.517695 211.069752 \n",
       "L 85.806143 214.087543 \n",
       "L 88.094592 216.6063 \n",
       "L 90.38304 218.710096 \n",
       "L 92.671488 220.468626 \n",
       "L 94.959937 221.939712 \n",
       "L 97.248385 223.171348 \n",
       "L 100.29965 224.508961 \n",
       "L 103.350914 225.567383 \n",
       "L 106.402179 226.406362 \n",
       "L 110.216259 227.216744 \n",
       "L 114.793156 227.92889 \n",
       "L 120.132869 228.507102 \n",
       "L 126.998214 228.988608 \n",
       "L 135.389192 229.336566 \n",
       "L 147.59425 229.597062 \n",
       "L 167.427469 229.765407 \n",
       "L 210.145173 229.85386 \n",
       "L 358.894318 229.874377 \n",
       "L 358.894318 229.874377 \n",
       "\" style=\"fill:none;stroke:#c44e52;stroke-linecap:round;stroke-width:2;\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_3\">\n",
       "    <path clip-path=\"url(#pd7e5f4e543)\" d=\"M 54.530682 35.031727 \n",
       "L 57.581946 50.401081 \n",
       "L 60.633211 64.529292 \n",
       "L 63.684475 77.51917 \n",
       "L 66.73574 89.464741 \n",
       "L 69.787004 100.45203 \n",
       "L 72.838269 110.559766 \n",
       "L 75.889533 119.860023 \n",
       "L 78.940798 128.418802 \n",
       "L 81.992063 136.29656 \n",
       "L 85.043327 143.548695 \n",
       "L 88.094592 150.225978 \n",
       "L 91.145856 156.374957 \n",
       "L 94.197121 162.038314 \n",
       "L 97.248385 167.255203 \n",
       "L 100.29965 172.061542 \n",
       "L 103.350914 176.490296 \n",
       "L 106.402179 180.571721 \n",
       "L 109.453443 184.333598 \n",
       "L 112.504708 187.801436 \n",
       "L 116.318788 191.758253 \n",
       "L 120.132869 195.333724 \n",
       "L 123.94695 198.565296 \n",
       "L 127.76103 201.486665 \n",
       "L 131.575111 204.128168 \n",
       "L 135.389192 206.517118 \n",
       "L 139.203272 208.678117 \n",
       "L 143.780169 211.001471 \n",
       "L 148.357066 213.062552 \n",
       "L 152.933963 214.891532 \n",
       "L 158.273676 216.767362 \n",
       "L 163.613389 218.400436 \n",
       "L 169.715918 220.010462 \n",
       "L 175.818447 221.386169 \n",
       "L 182.683792 222.696914 \n",
       "L 190.311953 223.90751 \n",
       "L 198.702931 224.99431 \n",
       "L 207.856724 225.944727 \n",
       "L 218.53615 226.81036 \n",
       "L 230.741208 227.555895 \n",
       "L 244.471898 228.167014 \n",
       "L 261.253853 228.685454 \n",
       "L 282.612705 229.108556 \n",
       "L 310.836902 229.429628 \n",
       "L 352.028973 229.65707 \n",
       "L 358.894318 229.679916 \n",
       "L 358.894318 229.679916 \n",
       "\" style=\"fill:none;stroke:#55a868;stroke-linecap:round;stroke-width:2;\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_4\">\n",
       "    <path clip-path=\"url(#pd7e5f4e543)\" d=\"M 54.530682 69.551667 \n",
       "L 55.293498 100.471294 \n",
       "L 56.056314 125.295611 \n",
       "L 56.81913 145.253818 \n",
       "L 57.581946 161.320458 \n",
       "L 58.344762 174.269934 \n",
       "L 59.107579 184.718892 \n",
       "L 59.870395 193.159301 \n",
       "L 60.633211 199.984343 \n",
       "L 61.396027 205.508752 \n",
       "L 62.158843 209.98483 \n",
       "L 62.921659 213.615103 \n",
       "L 63.684475 216.562335 \n",
       "L 64.447292 218.957478 \n",
       "L 65.210108 220.905999 \n",
       "L 65.972924 222.492919 \n",
       "L 66.73574 223.786833 \n",
       "L 67.498556 224.843131 \n",
       "L 69.024188 226.413359 \n",
       "L 70.549821 227.468548 \n",
       "L 72.075453 228.182399 \n",
       "L 73.601085 228.669101 \n",
       "L 75.889533 229.130671 \n",
       "L 78.940798 229.463351 \n",
       "L 83.517695 229.68699 \n",
       "L 91.908672 229.818185 \n",
       "L 114.793156 229.87102 \n",
       "L 358.894318 229.874489 \n",
       "L 358.894318 229.874489 \n",
       "\" style=\"fill:none;stroke:#4c72b0;stroke-linecap:round;stroke-width:2;\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_5\">\n",
       "    <path clip-path=\"url(#pd7e5f4e543)\" d=\"M 54.530682 133.869777 \n",
       "L 55.293498 182.668973 \n",
       "L 56.056314 206.264931 \n",
       "L 56.81913 217.844302 \n",
       "L 57.581946 223.603487 \n",
       "L 58.344762 226.510189 \n",
       "L 59.107579 228.004073 \n",
       "L 59.870395 228.790151 \n",
       "L 60.633211 229.216533 \n",
       "L 62.158843 229.597809 \n",
       "L 64.447292 229.778151 \n",
       "L 69.787004 229.862307 \n",
       "L 100.29965 229.874488 \n",
       "L 358.894318 229.874489 \n",
       "L 358.894318 229.874489 \n",
       "\" style=\"fill:none;stroke:#ccb974;stroke-linecap:round;stroke-width:2;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 39.3125 239.758125 \n",
       "L 39.3125 22.318125 \n",
       "\" style=\"fill:none;stroke:#262626;stroke-linecap:square;stroke-linejoin:miter;stroke-width:1.25;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 374.1125 239.758125 \n",
       "L 374.1125 22.318125 \n",
       "\" style=\"fill:none;stroke:#262626;stroke-linecap:square;stroke-linejoin:miter;stroke-width:1.25;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 39.3125 239.758125 \n",
       "L 374.1125 239.758125 \n",
       "\" style=\"fill:none;stroke:#262626;stroke-linecap:square;stroke-linejoin:miter;stroke-width:1.25;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 39.3125 22.318125 \n",
       "L 374.1125 22.318125 \n",
       "\" style=\"fill:none;stroke:#262626;stroke-linecap:square;stroke-linejoin:miter;stroke-width:1.25;\"/>\n",
       "   </g>\n",
       "   <g id=\"text_20\">\n",
       "    <!-- Convergence Graph with multiple alphas -->\n",
       "    <defs>\n",
       "     <path d=\"M 2.984375 54.6875 \n",
       "L 12.5 54.6875 \n",
       "L 29.59375 8.796875 \n",
       "L 46.6875 54.6875 \n",
       "L 56.203125 54.6875 \n",
       "L 35.6875 0 \n",
       "L 23.484375 0 \n",
       "z\n",
       "\" id=\"DejaVuSans-118\"/>\n",
       "     <path d=\"M 45.40625 27.984375 \n",
       "Q 45.40625 37.75 41.375 43.109375 \n",
       "Q 37.359375 48.484375 30.078125 48.484375 \n",
       "Q 22.859375 48.484375 18.828125 43.109375 \n",
       "Q 14.796875 37.75 14.796875 27.984375 \n",
       "Q 14.796875 18.265625 18.828125 12.890625 \n",
       "Q 22.859375 7.515625 30.078125 7.515625 \n",
       "Q 37.359375 7.515625 41.375 12.890625 \n",
       "Q 45.40625 18.265625 45.40625 27.984375 \n",
       "z\n",
       "M 54.390625 6.78125 \n",
       "Q 54.390625 -7.171875 48.1875 -13.984375 \n",
       "Q 42 -20.796875 29.203125 -20.796875 \n",
       "Q 24.46875 -20.796875 20.265625 -20.09375 \n",
       "Q 16.0625 -19.390625 12.109375 -17.921875 \n",
       "L 12.109375 -9.1875 \n",
       "Q 16.0625 -11.328125 19.921875 -12.34375 \n",
       "Q 23.78125 -13.375 27.78125 -13.375 \n",
       "Q 36.625 -13.375 41.015625 -8.765625 \n",
       "Q 45.40625 -4.15625 45.40625 5.171875 \n",
       "L 45.40625 9.625 \n",
       "Q 42.625 4.78125 38.28125 2.390625 \n",
       "Q 33.9375 0 27.875 0 \n",
       "Q 17.828125 0 11.671875 7.65625 \n",
       "Q 5.515625 15.328125 5.515625 27.984375 \n",
       "Q 5.515625 40.671875 11.671875 48.328125 \n",
       "Q 17.828125 56 27.875 56 \n",
       "Q 33.9375 56 38.28125 53.609375 \n",
       "Q 42.625 51.21875 45.40625 46.390625 \n",
       "L 45.40625 54.6875 \n",
       "L 54.390625 54.6875 \n",
       "z\n",
       "\" id=\"DejaVuSans-103\"/>\n",
       "     <path d=\"M 48.78125 52.59375 \n",
       "L 48.78125 44.1875 \n",
       "Q 44.96875 46.296875 41.140625 47.34375 \n",
       "Q 37.3125 48.390625 33.40625 48.390625 \n",
       "Q 24.65625 48.390625 19.8125 42.84375 \n",
       "Q 14.984375 37.3125 14.984375 27.296875 \n",
       "Q 14.984375 17.28125 19.8125 11.734375 \n",
       "Q 24.65625 6.203125 33.40625 6.203125 \n",
       "Q 37.3125 6.203125 41.140625 7.25 \n",
       "Q 44.96875 8.296875 48.78125 10.40625 \n",
       "L 48.78125 2.09375 \n",
       "Q 45.015625 0.34375 40.984375 -0.53125 \n",
       "Q 36.96875 -1.421875 32.421875 -1.421875 \n",
       "Q 20.0625 -1.421875 12.78125 6.34375 \n",
       "Q 5.515625 14.109375 5.515625 27.296875 \n",
       "Q 5.515625 40.671875 12.859375 48.328125 \n",
       "Q 20.21875 56 33.015625 56 \n",
       "Q 37.15625 56 41.109375 55.140625 \n",
       "Q 45.0625 54.296875 48.78125 52.59375 \n",
       "z\n",
       "\" id=\"DejaVuSans-99\"/>\n",
       "     <path d=\"M 59.515625 10.40625 \n",
       "L 59.515625 29.984375 \n",
       "L 43.40625 29.984375 \n",
       "L 43.40625 38.09375 \n",
       "L 69.28125 38.09375 \n",
       "L 69.28125 6.78125 \n",
       "Q 63.578125 2.734375 56.6875 0.65625 \n",
       "Q 49.8125 -1.421875 42 -1.421875 \n",
       "Q 24.90625 -1.421875 15.25 8.5625 \n",
       "Q 5.609375 18.5625 5.609375 36.375 \n",
       "Q 5.609375 54.25 15.25 64.234375 \n",
       "Q 24.90625 74.21875 42 74.21875 \n",
       "Q 49.125 74.21875 55.546875 72.453125 \n",
       "Q 61.96875 70.703125 67.390625 67.28125 \n",
       "L 67.390625 56.78125 \n",
       "Q 61.921875 61.421875 55.765625 63.765625 \n",
       "Q 49.609375 66.109375 42.828125 66.109375 \n",
       "Q 29.4375 66.109375 22.71875 58.640625 \n",
       "Q 16.015625 51.171875 16.015625 36.375 \n",
       "Q 16.015625 21.625 22.71875 14.15625 \n",
       "Q 29.4375 6.6875 42.828125 6.6875 \n",
       "Q 48.046875 6.6875 52.140625 7.59375 \n",
       "Q 56.25 8.5 59.515625 10.40625 \n",
       "z\n",
       "\" id=\"DejaVuSans-71\"/>\n",
       "     <path d=\"M 18.109375 8.203125 \n",
       "L 18.109375 -20.796875 \n",
       "L 9.078125 -20.796875 \n",
       "L 9.078125 54.6875 \n",
       "L 18.109375 54.6875 \n",
       "L 18.109375 46.390625 \n",
       "Q 20.953125 51.265625 25.265625 53.625 \n",
       "Q 29.59375 56 35.59375 56 \n",
       "Q 45.5625 56 51.78125 48.09375 \n",
       "Q 58.015625 40.1875 58.015625 27.296875 \n",
       "Q 58.015625 14.40625 51.78125 6.484375 \n",
       "Q 45.5625 -1.421875 35.59375 -1.421875 \n",
       "Q 29.59375 -1.421875 25.265625 0.953125 \n",
       "Q 20.953125 3.328125 18.109375 8.203125 \n",
       "z\n",
       "M 48.6875 27.296875 \n",
       "Q 48.6875 37.203125 44.609375 42.84375 \n",
       "Q 40.53125 48.484375 33.40625 48.484375 \n",
       "Q 26.265625 48.484375 22.1875 42.84375 \n",
       "Q 18.109375 37.203125 18.109375 27.296875 \n",
       "Q 18.109375 17.390625 22.1875 11.75 \n",
       "Q 26.265625 6.109375 33.40625 6.109375 \n",
       "Q 40.53125 6.109375 44.609375 11.75 \n",
       "Q 48.6875 17.390625 48.6875 27.296875 \n",
       "z\n",
       "\" id=\"DejaVuSans-112\"/>\n",
       "     <path d=\"M 54.890625 33.015625 \n",
       "L 54.890625 0 \n",
       "L 45.90625 0 \n",
       "L 45.90625 32.71875 \n",
       "Q 45.90625 40.484375 42.875 44.328125 \n",
       "Q 39.84375 48.1875 33.796875 48.1875 \n",
       "Q 26.515625 48.1875 22.3125 43.546875 \n",
       "Q 18.109375 38.921875 18.109375 30.90625 \n",
       "L 18.109375 0 \n",
       "L 9.078125 0 \n",
       "L 9.078125 75.984375 \n",
       "L 18.109375 75.984375 \n",
       "L 18.109375 46.1875 \n",
       "Q 21.34375 51.125 25.703125 53.5625 \n",
       "Q 30.078125 56 35.796875 56 \n",
       "Q 45.21875 56 50.046875 50.171875 \n",
       "Q 54.890625 44.34375 54.890625 33.015625 \n",
       "z\n",
       "\" id=\"DejaVuSans-104\"/>\n",
       "     <path d=\"M 4.203125 54.6875 \n",
       "L 13.1875 54.6875 \n",
       "L 24.421875 12.015625 \n",
       "L 35.59375 54.6875 \n",
       "L 46.1875 54.6875 \n",
       "L 57.421875 12.015625 \n",
       "L 68.609375 54.6875 \n",
       "L 77.59375 54.6875 \n",
       "L 63.28125 0 \n",
       "L 52.6875 0 \n",
       "L 40.921875 44.828125 \n",
       "L 29.109375 0 \n",
       "L 18.5 0 \n",
       "z\n",
       "\" id=\"DejaVuSans-119\"/>\n",
       "     <path d=\"M 9.421875 75.984375 \n",
       "L 18.40625 75.984375 \n",
       "L 18.40625 0 \n",
       "L 9.421875 0 \n",
       "z\n",
       "\" id=\"DejaVuSans-108\"/>\n",
       "    </defs>\n",
       "    <g style=\"fill:#262626;\" transform=\"translate(84.0275 16.318125)scale(0.12 -0.12)\">\n",
       "     <use xlink:href=\"#DejaVuSans-67\"/>\n",
       "     <use x=\"69.824219\" xlink:href=\"#DejaVuSans-111\"/>\n",
       "     <use x=\"131.005859\" xlink:href=\"#DejaVuSans-110\"/>\n",
       "     <use x=\"194.384766\" xlink:href=\"#DejaVuSans-118\"/>\n",
       "     <use x=\"253.564453\" xlink:href=\"#DejaVuSans-101\"/>\n",
       "     <use x=\"315.087891\" xlink:href=\"#DejaVuSans-114\"/>\n",
       "     <use x=\"356.185547\" xlink:href=\"#DejaVuSans-103\"/>\n",
       "     <use x=\"419.662109\" xlink:href=\"#DejaVuSans-101\"/>\n",
       "     <use x=\"481.185547\" xlink:href=\"#DejaVuSans-110\"/>\n",
       "     <use x=\"544.564453\" xlink:href=\"#DejaVuSans-99\"/>\n",
       "     <use x=\"599.544922\" xlink:href=\"#DejaVuSans-101\"/>\n",
       "     <use x=\"661.068359\" xlink:href=\"#DejaVuSans-32\"/>\n",
       "     <use x=\"692.855469\" xlink:href=\"#DejaVuSans-71\"/>\n",
       "     <use x=\"770.345703\" xlink:href=\"#DejaVuSans-114\"/>\n",
       "     <use x=\"811.458984\" xlink:href=\"#DejaVuSans-97\"/>\n",
       "     <use x=\"872.738281\" xlink:href=\"#DejaVuSans-112\"/>\n",
       "     <use x=\"936.214844\" xlink:href=\"#DejaVuSans-104\"/>\n",
       "     <use x=\"999.59375\" xlink:href=\"#DejaVuSans-32\"/>\n",
       "     <use x=\"1031.380859\" xlink:href=\"#DejaVuSans-119\"/>\n",
       "     <use x=\"1113.167969\" xlink:href=\"#DejaVuSans-105\"/>\n",
       "     <use x=\"1140.951172\" xlink:href=\"#DejaVuSans-116\"/>\n",
       "     <use x=\"1180.160156\" xlink:href=\"#DejaVuSans-104\"/>\n",
       "     <use x=\"1243.539062\" xlink:href=\"#DejaVuSans-32\"/>\n",
       "     <use x=\"1275.326172\" xlink:href=\"#DejaVuSans-109\"/>\n",
       "     <use x=\"1372.738281\" xlink:href=\"#DejaVuSans-117\"/>\n",
       "     <use x=\"1436.117188\" xlink:href=\"#DejaVuSans-108\"/>\n",
       "     <use x=\"1463.900391\" xlink:href=\"#DejaVuSans-116\"/>\n",
       "     <use x=\"1503.109375\" xlink:href=\"#DejaVuSans-105\"/>\n",
       "     <use x=\"1530.892578\" xlink:href=\"#DejaVuSans-112\"/>\n",
       "     <use x=\"1594.369141\" xlink:href=\"#DejaVuSans-108\"/>\n",
       "     <use x=\"1622.152344\" xlink:href=\"#DejaVuSans-101\"/>\n",
       "     <use x=\"1683.675781\" xlink:href=\"#DejaVuSans-32\"/>\n",
       "     <use x=\"1715.462891\" xlink:href=\"#DejaVuSans-97\"/>\n",
       "     <use x=\"1776.742188\" xlink:href=\"#DejaVuSans-108\"/>\n",
       "     <use x=\"1804.525391\" xlink:href=\"#DejaVuSans-112\"/>\n",
       "     <use x=\"1868.001953\" xlink:href=\"#DejaVuSans-104\"/>\n",
       "     <use x=\"1931.380859\" xlink:href=\"#DejaVuSans-97\"/>\n",
       "     <use x=\"1992.660156\" xlink:href=\"#DejaVuSans-115\"/>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"legend_1\">\n",
       "    <g id=\"patch_7\">\n",
       "     <path d=\"M 253.019687 111.847813 \n",
       "L 366.4125 111.847813 \n",
       "Q 368.6125 111.847813 368.6125 109.647813 \n",
       "L 368.6125 30.018125 \n",
       "Q 368.6125 27.818125 366.4125 27.818125 \n",
       "L 253.019687 27.818125 \n",
       "Q 250.819687 27.818125 250.819687 30.018125 \n",
       "L 250.819687 109.647813 \n",
       "Q 250.819687 111.847813 253.019687 111.847813 \n",
       "z\n",
       "\" style=\"fill:#ffffff;opacity:0.8;stroke:#cccccc;stroke-linejoin:miter;\"/>\n",
       "    </g>\n",
       "    <g id=\"line2d_6\">\n",
       "     <path d=\"M 255.219687 36.726406 \n",
       "L 277.219687 36.726406 \n",
       "\" style=\"fill:none;stroke:#64b5cd;stroke-linecap:round;stroke-width:2;\"/>\n",
       "    </g>\n",
       "    <g id=\"line2d_7\"/>\n",
       "    <g id=\"text_21\">\n",
       "     <!-- alpha = 0.003 -->\n",
       "     <defs>\n",
       "      <path d=\"M 10.59375 45.40625 \n",
       "L 73.1875 45.40625 \n",
       "L 73.1875 37.203125 \n",
       "L 10.59375 37.203125 \n",
       "z\n",
       "M 10.59375 25.484375 \n",
       "L 73.1875 25.484375 \n",
       "L 73.1875 17.1875 \n",
       "L 10.59375 17.1875 \n",
       "z\n",
       "\" id=\"DejaVuSans-61\"/>\n",
       "      <path d=\"M 10.6875 12.40625 \n",
       "L 21 12.40625 \n",
       "L 21 0 \n",
       "L 10.6875 0 \n",
       "z\n",
       "\" id=\"DejaVuSans-46\"/>\n",
       "     </defs>\n",
       "     <g style=\"fill:#262626;\" transform=\"translate(286.019687 40.576406)scale(0.11 -0.11)\">\n",
       "      <use xlink:href=\"#DejaVuSans-97\"/>\n",
       "      <use x=\"61.279297\" xlink:href=\"#DejaVuSans-108\"/>\n",
       "      <use x=\"89.0625\" xlink:href=\"#DejaVuSans-112\"/>\n",
       "      <use x=\"152.539062\" xlink:href=\"#DejaVuSans-104\"/>\n",
       "      <use x=\"215.917969\" xlink:href=\"#DejaVuSans-97\"/>\n",
       "      <use x=\"277.197266\" xlink:href=\"#DejaVuSans-32\"/>\n",
       "      <use x=\"308.984375\" xlink:href=\"#DejaVuSans-61\"/>\n",
       "      <use x=\"392.773438\" xlink:href=\"#DejaVuSans-32\"/>\n",
       "      <use x=\"424.560547\" xlink:href=\"#DejaVuSans-48\"/>\n",
       "      <use x=\"488.183594\" xlink:href=\"#DejaVuSans-46\"/>\n",
       "      <use x=\"519.970703\" xlink:href=\"#DejaVuSans-48\"/>\n",
       "      <use x=\"583.59375\" xlink:href=\"#DejaVuSans-48\"/>\n",
       "      <use x=\"647.216797\" xlink:href=\"#DejaVuSans-51\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"line2d_8\">\n",
       "     <path d=\"M 255.219687 52.872344 \n",
       "L 277.219687 52.872344 \n",
       "\" style=\"fill:none;stroke:#c44e52;stroke-linecap:round;stroke-width:2;\"/>\n",
       "    </g>\n",
       "    <g id=\"line2d_9\"/>\n",
       "    <g id=\"text_22\">\n",
       "     <!-- alpha = 0.03 -->\n",
       "     <g style=\"fill:#262626;\" transform=\"translate(286.019687 56.722344)scale(0.11 -0.11)\">\n",
       "      <use xlink:href=\"#DejaVuSans-97\"/>\n",
       "      <use x=\"61.279297\" xlink:href=\"#DejaVuSans-108\"/>\n",
       "      <use x=\"89.0625\" xlink:href=\"#DejaVuSans-112\"/>\n",
       "      <use x=\"152.539062\" xlink:href=\"#DejaVuSans-104\"/>\n",
       "      <use x=\"215.917969\" xlink:href=\"#DejaVuSans-97\"/>\n",
       "      <use x=\"277.197266\" xlink:href=\"#DejaVuSans-32\"/>\n",
       "      <use x=\"308.984375\" xlink:href=\"#DejaVuSans-61\"/>\n",
       "      <use x=\"392.773438\" xlink:href=\"#DejaVuSans-32\"/>\n",
       "      <use x=\"424.560547\" xlink:href=\"#DejaVuSans-48\"/>\n",
       "      <use x=\"488.183594\" xlink:href=\"#DejaVuSans-46\"/>\n",
       "      <use x=\"519.970703\" xlink:href=\"#DejaVuSans-48\"/>\n",
       "      <use x=\"583.59375\" xlink:href=\"#DejaVuSans-51\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"line2d_10\">\n",
       "     <path d=\"M 255.219687 69.018281 \n",
       "L 277.219687 69.018281 \n",
       "\" style=\"fill:none;stroke:#55a868;stroke-linecap:round;stroke-width:2;\"/>\n",
       "    </g>\n",
       "    <g id=\"line2d_11\"/>\n",
       "    <g id=\"text_23\">\n",
       "     <!-- alpha = 0.01 -->\n",
       "     <g style=\"fill:#262626;\" transform=\"translate(286.019687 72.868281)scale(0.11 -0.11)\">\n",
       "      <use xlink:href=\"#DejaVuSans-97\"/>\n",
       "      <use x=\"61.279297\" xlink:href=\"#DejaVuSans-108\"/>\n",
       "      <use x=\"89.0625\" xlink:href=\"#DejaVuSans-112\"/>\n",
       "      <use x=\"152.539062\" xlink:href=\"#DejaVuSans-104\"/>\n",
       "      <use x=\"215.917969\" xlink:href=\"#DejaVuSans-97\"/>\n",
       "      <use x=\"277.197266\" xlink:href=\"#DejaVuSans-32\"/>\n",
       "      <use x=\"308.984375\" xlink:href=\"#DejaVuSans-61\"/>\n",
       "      <use x=\"392.773438\" xlink:href=\"#DejaVuSans-32\"/>\n",
       "      <use x=\"424.560547\" xlink:href=\"#DejaVuSans-48\"/>\n",
       "      <use x=\"488.183594\" xlink:href=\"#DejaVuSans-46\"/>\n",
       "      <use x=\"519.970703\" xlink:href=\"#DejaVuSans-48\"/>\n",
       "      <use x=\"583.59375\" xlink:href=\"#DejaVuSans-49\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"line2d_12\">\n",
       "     <path d=\"M 255.219687 85.164219 \n",
       "L 277.219687 85.164219 \n",
       "\" style=\"fill:none;stroke:#4c72b0;stroke-linecap:round;stroke-width:2;\"/>\n",
       "    </g>\n",
       "    <g id=\"line2d_13\"/>\n",
       "    <g id=\"text_24\">\n",
       "     <!-- alpha = 0.1 -->\n",
       "     <g style=\"fill:#262626;\" transform=\"translate(286.019687 89.014219)scale(0.11 -0.11)\">\n",
       "      <use xlink:href=\"#DejaVuSans-97\"/>\n",
       "      <use x=\"61.279297\" xlink:href=\"#DejaVuSans-108\"/>\n",
       "      <use x=\"89.0625\" xlink:href=\"#DejaVuSans-112\"/>\n",
       "      <use x=\"152.539062\" xlink:href=\"#DejaVuSans-104\"/>\n",
       "      <use x=\"215.917969\" xlink:href=\"#DejaVuSans-97\"/>\n",
       "      <use x=\"277.197266\" xlink:href=\"#DejaVuSans-32\"/>\n",
       "      <use x=\"308.984375\" xlink:href=\"#DejaVuSans-61\"/>\n",
       "      <use x=\"392.773438\" xlink:href=\"#DejaVuSans-32\"/>\n",
       "      <use x=\"424.560547\" xlink:href=\"#DejaVuSans-48\"/>\n",
       "      <use x=\"488.183594\" xlink:href=\"#DejaVuSans-46\"/>\n",
       "      <use x=\"519.970703\" xlink:href=\"#DejaVuSans-49\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"line2d_14\">\n",
       "     <path d=\"M 255.219687 101.310156 \n",
       "L 277.219687 101.310156 \n",
       "\" style=\"fill:none;stroke:#ccb974;stroke-linecap:round;stroke-width:2;\"/>\n",
       "    </g>\n",
       "    <g id=\"line2d_15\"/>\n",
       "    <g id=\"text_25\">\n",
       "     <!-- alpha = 0.3 -->\n",
       "     <g style=\"fill:#262626;\" transform=\"translate(286.019687 105.160156)scale(0.11 -0.11)\">\n",
       "      <use xlink:href=\"#DejaVuSans-97\"/>\n",
       "      <use x=\"61.279297\" xlink:href=\"#DejaVuSans-108\"/>\n",
       "      <use x=\"89.0625\" xlink:href=\"#DejaVuSans-112\"/>\n",
       "      <use x=\"152.539062\" xlink:href=\"#DejaVuSans-104\"/>\n",
       "      <use x=\"215.917969\" xlink:href=\"#DejaVuSans-97\"/>\n",
       "      <use x=\"277.197266\" xlink:href=\"#DejaVuSans-32\"/>\n",
       "      <use x=\"308.984375\" xlink:href=\"#DejaVuSans-61\"/>\n",
       "      <use x=\"392.773438\" xlink:href=\"#DejaVuSans-32\"/>\n",
       "      <use x=\"424.560547\" xlink:href=\"#DejaVuSans-48\"/>\n",
       "      <use x=\"488.183594\" xlink:href=\"#DejaVuSans-46\"/>\n",
       "      <use x=\"519.970703\" xlink:href=\"#DejaVuSans-51\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"pd7e5f4e543\">\n",
       "   <rect height=\"217.44\" width=\"334.8\" x=\"39.3125\" y=\"22.318125\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# runnning gradient descend with different alphas\n",
    "theta1, J_history1 = gradient_descent(X, y, initial_theta, 0.01, num_iters)\n",
    "theta2, J_history2 = gradient_descent(X, y, initial_theta, 0.1, num_iters)\n",
    "theta3, J_history3 = gradient_descent(X, y, initial_theta, 0.3, num_iters)\n",
    "theta4, J_history4 = gradient_descent(X, y, initial_theta, 0.003, num_iters)\n",
    "\n",
    "# Plot the convergence graph\n",
    "fig = plt.figure(\"Covariance Graph\")\n",
    "ax = fig.subplots()\n",
    "ax.set_title(\"Convergence Graph with multiple alphas\")\n",
    "ax.plot(range(J_history4.size), J_history4, lw=2,c='c', label=\"alpha = 0.003\")\n",
    "ax.plot(range(J_history.size), J_history, lw=2, c='r', label=\"alpha = 0.03\")\n",
    "ax.plot(range(J_history1.size), J_history1, lw=2, c='g' , label=\"alpha = 0.01\")\n",
    "ax.plot(range(J_history2.size), J_history2, lw=2, c='b',  label=\"alpha = 0.1\")\n",
    "ax.plot(range(J_history3.size), J_history3, lw=2, c='y',  label=\"alpha = 0.3\")\n",
    "\n",
    "ax.legend()\n",
    "ax.set_xlabel('Number of iterations')\n",
    "ax.set_ylabel('Cost J')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice the changes in the convergence curves as the learning rate changes.\n",
    "With a small learning rate, you should find that gradient descent takes a very\n",
    "long time to converge to the optimal value. Conversely, with a large learning\n",
    "rate, gradient descent might not converge or might even diverge!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Theta computed from gradient descent: \n",
      "[340410.91897274 109162.68848142  -6293.24735132]\n",
      "\n",
      "Predicted price of a 1650 sq-ft, 3 br house (using gradient descent): 293142.43\n"
     ]
    }
   ],
   "source": [
    "# Display gradient descent's result\n",
    "print('Theta computed from gradient descent: ')\n",
    "print(theta.ravel())\n",
    "\n",
    "# Estimate the price of a 1650 sq-ft, 3 br house\n",
    "\n",
    "test_data = np.array([1650,3]).reshape(1, 2)\n",
    "test_data, _, __ = feature_normalize(test_data, mu, sigma)\n",
    "test_data = add_bias_unit(test_data)\n",
    "price = test_data.dot(theta)\n",
    "# ============================================================\n",
    "\n",
    "print('\\nPredicted price of a 1650 sq-ft, 3 br house (using gradient descent): {:.2f}'.format(price[0,0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extra (Normal Equation):\n",
    "\n",
    "In **Normal Equation** method, we will minimize J by explicitly taking its derivatives with respect to $\\theta_{j}$'s amd setting them to zero. This allows us to find the optimum theta withut iterations.\n",
    "\n",
    "In case of small datasets it is preferable to use normal equation to find the minimum theta values.\n",
    "\n",
    "Key feature of Normal Equation are:\n",
    "- No need to choose learning rate alpha.\n",
    "- No need of iterations.\n",
    "- better option when the number of features is small.\n",
    "- No need of feature scaling or normalization.\n",
    "\n",
    "The vectorized formula for normal equation is:\n",
    "\n",
    "$$ \\theta = \\left( X^{T}X \\right)^{-1}X^{T}\\vec{y}$$\n",
    "\n",
    "where:\n",
    "$X^{T}$ is the transpose of matrix X.\n",
    "\n",
    "Note: the X matrix should contain the $x_{0} = 1$ bias unit in each data point."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading data ...\n",
      "Solving with normal equations...\n",
      "Theta computed from the normal equations: \n",
      "[89597.9095428    139.21067402 -8738.01911233]\n",
      "Predicted price of a 1650 sq-ft, 3 br house (using normal equations): 293081.46\n"
     ]
    }
   ],
   "source": [
    "# ================Normal Equations ================\n",
    "\n",
    "print('Loading data ...')\n",
    "\n",
    "data = np.loadtxt('data/ex1data2.txt', delimiter=',')\n",
    "X = data[:, :-1]  # 47x2\n",
    "y = data[:, -1, None]  # 47x1\n",
    "m = y.size  # 47\n",
    "\n",
    "# adding bias unit to data\n",
    "X = add_bias_unit(X)\n",
    "\n",
    "print('Solving with normal equations...')\n",
    "\n",
    "# Calculate the parameters from the normal equation\n",
    "theta_norm = normal_equation(X, y)\n",
    "\n",
    "# Display normal equation's result\n",
    "print('Theta computed from the normal equations: ')\n",
    "print(theta_norm.ravel())\n",
    "\n",
    "# Estimate the price of a 1650 sq-ft, 3 br house\n",
    "test_data1 = np.array([1650,3]).reshape(1, 2)\n",
    "test_data1 = add_bias_unit(test_data1)\n",
    "price_norm = test_data1.dot(theta_norm)\n",
    "\n",
    "print('Predicted price of a 1650 sq-ft, 3 br house (using normal equations): {:.2f}'.format(price_norm[0,0]))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}